[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260808 01:08:28.434537 32593 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.31.212.126:39845
I20260808 01:08:28.436774 32593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:08:28.438119 32593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 01:08:28.472522 32603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:08:28.472561 32600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:08:28.473124 32601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:08:29.787225 32602 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260808 01:08:29.787397 32593 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:08:29.792269 32593 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:08:29.792562 32593 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:08:29.792717 32593 hybrid_clock.cc:648] HybridClock initialized: now 1786151309792703 us; error 0 us; skew 500 ppm
I20260808 01:08:29.793942 32593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:08:29.800751 32593 webserver.cc:460] Webserver started at http://127.31.212.126:39687/ using document root <none> and password file <none>
I20260808 01:08:29.801827 32593 fs_manager.cc:359] Metadata directory not provided
I20260808 01:08:29.802057 32593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:08:29.802524 32593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:08:29.809522 32593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/master-0-root/instance:
uuid: "643de1b5abe34a92afee3d3ced1b7c6d"
format_stamp: "Formatted at 2026-08-08 01:08:29 on dist-test-slave-mvvj"
I20260808 01:08:29.819264 32593 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.003s
I20260808 01:08:29.826136 32610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:08:29.827934 32593 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260808 01:08:29.828301 32593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/master-0-root
uuid: "643de1b5abe34a92afee3d3ced1b7c6d"
format_stamp: "Formatted at 2026-08-08 01:08:29 on dist-test-slave-mvvj"
I20260808 01:08:29.828641 32593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:08:29.869818 32593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 01:08:29.871701 32593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:08:29.872272 32593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:08:29.965819 32593 rpc_server.cc:307] RPC server started. Bound to: 127.31.212.126:39845
I20260808 01:08:29.965894 32663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.212.126:39845 every 8 connection(s)
I20260808 01:08:29.975685 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:30.005169 32664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d: Bootstrap starting.
I20260808 01:08:30.012841 32664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:30.015466 32664 log.cc:824] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d: Log is configured to *not* fsync() on all Append() calls
I20260808 01:08:30.021538 32664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d: No bootstrap required, opened a new log
I20260808 01:08:30.049080 32664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643de1b5abe34a92afee3d3ced1b7c6d" member_type: VOTER }
I20260808 01:08:30.050004 32664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:30.050345 32664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643de1b5abe34a92afee3d3ced1b7c6d, State: Initialized, Role: FOLLOWER
I20260808 01:08:30.051668 32664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643de1b5abe34a92afee3d3ced1b7c6d" member_type: VOTER }
I20260808 01:08:30.052350 32664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 01:08:30.052714 32664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 01:08:30.053118 32664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:30.059602 32664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643de1b5abe34a92afee3d3ced1b7c6d" member_type: VOTER }
I20260808 01:08:30.061197 32664 leader_election.cc:302] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 643de1b5abe34a92afee3d3ced1b7c6d; no voters:
I20260808 01:08:30.063436 32664 leader_election.cc:288] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 01:08:30.063778 32667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:08:30.068264 32667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [term 1 LEADER]: Becoming Leader. State: Replica: 643de1b5abe34a92afee3d3ced1b7c6d, State: Running, Role: LEADER
I20260808 01:08:30.069646 32667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643de1b5abe34a92afee3d3ced1b7c6d" member_type: VOTER }
I20260808 01:08:30.070827 32664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 01:08:30.087301 32668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 643de1b5abe34a92afee3d3ced1b7c6d. Latest consensus state: current_term: 1 leader_uuid: "643de1b5abe34a92afee3d3ced1b7c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643de1b5abe34a92afee3d3ced1b7c6d" member_type: VOTER } }
I20260808 01:08:30.088899 32668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [sys.catalog]: This master's current role is: LEADER
I20260808 01:08:30.090050 32669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "643de1b5abe34a92afee3d3ced1b7c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643de1b5abe34a92afee3d3ced1b7c6d" member_type: VOTER } }
I20260808 01:08:30.091185 32669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [sys.catalog]: This master's current role is: LEADER
I20260808 01:08:30.095570 32677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 01:08:30.113417 32677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 01:08:30.116142 32593 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260808 01:08:30.135418 32677 catalog_manager.cc:1347] Generated new cluster ID: 9a8cc42470174855ad0b2e7f4aad7e02
I20260808 01:08:30.135802 32677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 01:08:30.202493 32677 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 01:08:30.205262 32677 catalog_manager.cc:1504] Loading token signing keys...
I20260808 01:08:30.219432 32677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d: Generated new TSK 0
I20260808 01:08:30.220849 32677 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 01:08:30.248889 32593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 01:08:30.262444 32687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:08:30.264505 32688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:08:30.271116 32690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:08:30.271240 32593 server_base.cc:1034] running on GCE node
I20260808 01:08:30.272493 32593 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:08:30.272723 32593 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:08:30.272910 32593 hybrid_clock.cc:648] HybridClock initialized: now 1786151310272888 us; error 0 us; skew 500 ppm
I20260808 01:08:30.273519 32593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:08:30.278784 32593 webserver.cc:460] Webserver started at http://127.31.212.65:45607/ using document root <none> and password file <none>
I20260808 01:08:30.279393 32593 fs_manager.cc:359] Metadata directory not provided
I20260808 01:08:30.279634 32593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:08:30.280009 32593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:08:30.281590 32593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/ts-0-root/instance:
uuid: "15218c77abfe47d9ab4765853cc3b2c4"
format_stamp: "Formatted at 2026-08-08 01:08:30 on dist-test-slave-mvvj"
I20260808 01:08:30.288233 32593 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 01:08:30.293660 32695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:08:30.294615 32593 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260808 01:08:30.294927 32593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/ts-0-root
uuid: "15218c77abfe47d9ab4765853cc3b2c4"
format_stamp: "Formatted at 2026-08-08 01:08:30 on dist-test-slave-mvvj"
I20260808 01:08:30.295250 32593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:08:30.332528 32593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 01:08:30.333923 32593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:08:30.338277 32593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 01:08:30.344030 32593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:08:30.344365 32593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:08:30.344659 32593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 01:08:30.344803 32593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:08:30.479409 32593 rpc_server.cc:307] RPC server started. Bound to: 127.31.212.65:40753
I20260808 01:08:30.479467 32760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.212.65:40753 every 8 connection(s)
I20260808 01:08:30.487325 32593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 01:08:30.501101 32765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:08:30.503285 32766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:08:30.514257 32593 server_base.cc:1034] running on GCE node
W20260808 01:08:30.516916 300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:08:30.518246 32593 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:08:30.518538 32593 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:08:30.518749 32593 hybrid_clock.cc:648] HybridClock initialized: now 1786151310518733 us; error 0 us; skew 500 ppm
I20260808 01:08:30.519642 32593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:08:30.524122 32761 heartbeater.cc:347] Connected to a master server at 127.31.212.126:39845
I20260808 01:08:30.524636 32761 heartbeater.cc:464] Registering TS with master...
I20260808 01:08:30.525871 32593 webserver.cc:460] Webserver started at http://127.31.212.66:41373/ using document root <none> and password file <none>
I20260808 01:08:30.525883 32761 heartbeater.cc:511] Master 127.31.212.126:39845 requested a full tablet report, sending...
I20260808 01:08:30.526695 32593 fs_manager.cc:359] Metadata directory not provided
I20260808 01:08:30.526993 32593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:08:30.527330 32593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:08:30.528671 32593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/ts-1-root/instance:
uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
format_stamp: "Formatted at 2026-08-08 01:08:30 on dist-test-slave-mvvj"
I20260808 01:08:30.530216 32627 ts_manager.cc:194] Registered new tserver with Master: 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753)
I20260808 01:08:30.533713 32627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51810
I20260808 01:08:30.535916 32593 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260808 01:08:30.540608 306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:08:30.541592 32593 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260808 01:08:30.541900 32593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/ts-1-root
uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
format_stamp: "Formatted at 2026-08-08 01:08:30 on dist-test-slave-mvvj"
I20260808 01:08:30.542215 32593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:08:30.569694 32593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 01:08:30.571208 32593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:08:30.574803 32593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 01:08:30.579228 32593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:08:30.579483 32593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:08:30.579798 32593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 01:08:30.580013 32593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260808 01:08:30.698016 32593 rpc_server.cc:307] RPC server started. Bound to: 127.31.212.66:34833
I20260808 01:08:30.698370 374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.212.66:34833 every 8 connection(s)
I20260808 01:08:30.708706 32593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 01:08:30.718405 378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:08:30.724960 375 heartbeater.cc:347] Connected to a master server at 127.31.212.126:39845
I20260808 01:08:30.725550 375 heartbeater.cc:464] Registering TS with master...
I20260808 01:08:30.726471 375 heartbeater.cc:511] Master 127.31.212.126:39845 requested a full tablet report, sending...
I20260808 01:08:30.729204 32627 ts_manager.cc:194] Registered new tserver with Master: 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
W20260808 01:08:30.729707 379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:08:30.730016 32593 server_base.cc:1034] running on GCE node
W20260808 01:08:30.730947 381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:08:30.732038 32593 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260808 01:08:30.731963 32627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51820
W20260808 01:08:30.732343 32593 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:08:30.732604 32593 hybrid_clock.cc:648] HybridClock initialized: now 1786151310732586 us; error 0 us; skew 500 ppm
I20260808 01:08:30.733343 32593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:08:30.736516 32593 webserver.cc:460] Webserver started at http://127.31.212.67:33131/ using document root <none> and password file <none>
I20260808 01:08:30.737066 32593 fs_manager.cc:359] Metadata directory not provided
I20260808 01:08:30.737354 32593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:08:30.737634 32593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:08:30.738760 32593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/ts-2-root/instance:
uuid: "90b6685012e44e47a191e25ffc352954"
format_stamp: "Formatted at 2026-08-08 01:08:30 on dist-test-slave-mvvj"
I20260808 01:08:30.743884 32593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260808 01:08:30.747545 386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:08:30.748462 32593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 01:08:30.748845 32593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/ts-2-root
uuid: "90b6685012e44e47a191e25ffc352954"
format_stamp: "Formatted at 2026-08-08 01:08:30 on dist-test-slave-mvvj"
I20260808 01:08:30.749202 32593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786151308190403-32593-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:08:30.770895 32593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 01:08:30.772536 32593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:08:30.775133 32593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 01:08:30.778502 32593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:08:30.778754 32593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:08:30.779026 32593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 01:08:30.779203 32593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:08:30.891825 32593 rpc_server.cc:307] RPC server started. Bound to: 127.31.212.67:42685
I20260808 01:08:30.892658 450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.212.67:42685 every 8 connection(s)
I20260808 01:08:30.913569 451 heartbeater.cc:347] Connected to a master server at 127.31.212.126:39845
I20260808 01:08:30.914033 451 heartbeater.cc:464] Registering TS with master...
I20260808 01:08:30.915011 451 heartbeater.cc:511] Master 127.31.212.126:39845 requested a full tablet report, sending...
I20260808 01:08:30.917536 32627 ts_manager.cc:194] Registered new tserver with Master: 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:30.917639 32593 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.019900949s
I20260808 01:08:30.919960 32627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51830
I20260808 01:08:30.922012 32593 test_util.cc:268] Using random seed: -1033432829
I20260808 01:08:30.984490 32627 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51834:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 01:08:30.989884 32627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260808 01:08:31.123008 32678 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:08:31.154608 413 tablet_service.cc:1467] Processing CreateTablet for tablet 26f5c989c2b7409c8ee0112f26936deb (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260808 01:08:31.155864 411 tablet_service.cc:1467] Processing CreateTablet for tablet bcbc69178831403785892df61938ee97 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260808 01:08:31.157418 415 tablet_service.cc:1467] Processing CreateTablet for tablet ea7e59a231e74d54b4859d7f52e8f0d1 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260808 01:08:31.154862 414 tablet_service.cc:1467] Processing CreateTablet for tablet d120a88b81634ee28ee1ec136a4d4143 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260808 01:08:31.156553 410 tablet_service.cc:1467] Processing CreateTablet for tablet 7da895fe9b014f5d8f2acede20d67c53 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260808 01:08:31.163548 408 tablet_service.cc:1467] Processing CreateTablet for tablet b0a4791d89904ee29ff78ef43c021b65 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260808 01:08:31.164397 409 tablet_service.cc:1467] Processing CreateTablet for tablet 6cda7aa230904dc6bff9da8fba188d58 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260808 01:08:31.171396 32725 tablet_service.cc:1467] Processing CreateTablet for tablet ea7e59a231e74d54b4859d7f52e8f0d1 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260808 01:08:31.177618 32723 tablet_service.cc:1467] Processing CreateTablet for tablet 26f5c989c2b7409c8ee0112f26936deb (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260808 01:08:31.184818 32720 tablet_service.cc:1467] Processing CreateTablet for tablet 7da895fe9b014f5d8f2acede20d67c53 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260808 01:08:31.174160 32724 tablet_service.cc:1467] Processing CreateTablet for tablet d120a88b81634ee28ee1ec136a4d4143 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260808 01:08:31.187252 32722 tablet_service.cc:1467] Processing CreateTablet for tablet bcbc69178831403785892df61938ee97 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260808 01:08:31.186264 32723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f5c989c2b7409c8ee0112f26936deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.189360 413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f5c989c2b7409c8ee0112f26936deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.181828 407 tablet_service.cc:1467] Processing CreateTablet for tablet 9f8a848bc0b4438b8ae3e5a5ff59f3b6 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260808 01:08:31.186528 32719 tablet_service.cc:1467] Processing CreateTablet for tablet 6cda7aa230904dc6bff9da8fba188d58 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260808 01:08:31.191015 406 tablet_service.cc:1467] Processing CreateTablet for tablet ca9ecb475d7143bd94ed0590fd4c5368 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260808 01:08:31.196779 32718 tablet_service.cc:1467] Processing CreateTablet for tablet b0a4791d89904ee29ff78ef43c021b65 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260808 01:08:31.200845 32717 tablet_service.cc:1467] Processing CreateTablet for tablet 9f8a848bc0b4438b8ae3e5a5ff59f3b6 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260808 01:08:31.204185 32724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d120a88b81634ee28ee1ec136a4d4143. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.216837 32717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f8a848bc0b4438b8ae3e5a5ff59f3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.221002 32716 tablet_service.cc:1467] Processing CreateTablet for tablet ca9ecb475d7143bd94ed0590fd4c5368 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260808 01:08:31.221056 32719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cda7aa230904dc6bff9da8fba188d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.226851 32725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7e59a231e74d54b4859d7f52e8f0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.233603 408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0a4791d89904ee29ff78ef43c021b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.234730 411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcbc69178831403785892df61938ee97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.235140 32720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da895fe9b014f5d8f2acede20d67c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.238600 32716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca9ecb475d7143bd94ed0590fd4c5368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.240315 32718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0a4791d89904ee29ff78ef43c021b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.237629 415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7e59a231e74d54b4859d7f52e8f0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.246258 409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cda7aa230904dc6bff9da8fba188d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.248371 406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca9ecb475d7143bd94ed0590fd4c5368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.250988 407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f8a848bc0b4438b8ae3e5a5ff59f3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.253863 32722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcbc69178831403785892df61938ee97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.265331 414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d120a88b81634ee28ee1ec136a4d4143. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.293104 410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da895fe9b014f5d8f2acede20d67c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.298723 338 tablet_service.cc:1467] Processing CreateTablet for tablet ea7e59a231e74d54b4859d7f52e8f0d1 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260808 01:08:31.330497 334 tablet_service.cc:1467] Processing CreateTablet for tablet bcbc69178831403785892df61938ee97 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260808 01:08:31.325150 337 tablet_service.cc:1467] Processing CreateTablet for tablet d120a88b81634ee28ee1ec136a4d4143 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260808 01:08:31.338795 332 tablet_service.cc:1467] Processing CreateTablet for tablet 7da895fe9b014f5d8f2acede20d67c53 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260808 01:08:31.340315 338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7e59a231e74d54b4859d7f52e8f0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.347642 336 tablet_service.cc:1467] Processing CreateTablet for tablet 26f5c989c2b7409c8ee0112f26936deb (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260808 01:08:31.359788 336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f5c989c2b7409c8ee0112f26936deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.364343 337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d120a88b81634ee28ee1ec136a4d4143. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.374135 334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcbc69178831403785892df61938ee97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.383589 332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da895fe9b014f5d8f2acede20d67c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.405819 331 tablet_service.cc:1467] Processing CreateTablet for tablet 6cda7aa230904dc6bff9da8fba188d58 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260808 01:08:31.410948 331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cda7aa230904dc6bff9da8fba188d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.433274 330 tablet_service.cc:1467] Processing CreateTablet for tablet b0a4791d89904ee29ff78ef43c021b65 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260808 01:08:31.436278 330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0a4791d89904ee29ff78ef43c021b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.470526 329 tablet_service.cc:1467] Processing CreateTablet for tablet 9f8a848bc0b4438b8ae3e5a5ff59f3b6 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260808 01:08:31.473176 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f8a848bc0b4438b8ae3e5a5ff59f3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.482488 469 tablet_bootstrap.cc:492] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954: Bootstrap starting.
I20260808 01:08:31.503690 327 tablet_service.cc:1467] Processing CreateTablet for tablet ca9ecb475d7143bd94ed0590fd4c5368 (DEFAULT_TABLE table=test-workload [id=925a226353e643b5a84bb88c13a640e9]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260808 01:08:31.512413 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca9ecb475d7143bd94ed0590fd4c5368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:08:31.538892 32761 heartbeater.cc:503] Master 127.31.212.126:39845 was elected leader, sending a full tablet report...
I20260808 01:08:31.546504 470 tablet_bootstrap.cc:492] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4: Bootstrap starting.
I20260808 01:08:31.620229 469 tablet_bootstrap.cc:654] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:31.645679 471 tablet_bootstrap.cc:492] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a: Bootstrap starting.
I20260808 01:08:31.653261 470 tablet_bootstrap.cc:654] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:31.673390 469 tablet_bootstrap.cc:492] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954: No bootstrap required, opened a new log
I20260808 01:08:31.673880 469 ts_tablet_manager.cc:1397] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954: Time spent bootstrapping tablet: real 0.203s user 0.010s sys 0.009s
I20260808 01:08:31.685678 470 tablet_bootstrap.cc:492] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4: No bootstrap required, opened a new log
I20260808 01:08:31.686687 470 ts_tablet_manager.cc:1397] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4: Time spent bootstrapping tablet: real 0.173s user 0.003s sys 0.019s
I20260808 01:08:31.687264 471 tablet_bootstrap.cc:654] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:31.686818 469 raft_consensus.cc:348] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.688262 469 raft_consensus.cc:374] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:31.688656 469 raft_consensus.cc:729] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Initialized, Role: FOLLOWER
I20260808 01:08:31.689805 469 consensus_queue.cc:260] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.691608 470 raft_consensus.cc:348] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.692759 470 raft_consensus.cc:374] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:31.693176 470 raft_consensus.cc:729] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Initialized, Role: FOLLOWER
I20260808 01:08:31.698913 451 heartbeater.cc:503] Master 127.31.212.126:39845 was elected leader, sending a full tablet report...
I20260808 01:08:31.698243 470 consensus_queue.cc:260] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.699994 469 ts_tablet_manager.cc:1428] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954: Time spent starting tablet: real 0.026s user 0.010s sys 0.007s
I20260808 01:08:31.701092 469 tablet_bootstrap.cc:492] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954: Bootstrap starting.
I20260808 01:08:31.706177 471 tablet_bootstrap.cc:492] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a: No bootstrap required, opened a new log
I20260808 01:08:31.708120 471 ts_tablet_manager.cc:1397] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent bootstrapping tablet: real 0.068s user 0.017s sys 0.007s
I20260808 01:08:31.709760 469 tablet_bootstrap.cc:654] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:31.711735 471 raft_consensus.cc:348] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:31.713057 471 raft_consensus.cc:374] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:31.713542 471 raft_consensus.cc:729] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Initialized, Role: FOLLOWER
I20260808 01:08:31.714958 471 consensus_queue.cc:260] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:31.724138 469 tablet_bootstrap.cc:492] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954: No bootstrap required, opened a new log
I20260808 01:08:31.724429 470 ts_tablet_manager.cc:1428] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4: Time spent starting tablet: real 0.037s user 0.010s sys 0.005s
I20260808 01:08:31.724756 469 ts_tablet_manager.cc:1397] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.007s
I20260808 01:08:31.728128 470 tablet_bootstrap.cc:492] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4: Bootstrap starting.
I20260808 01:08:31.728689 469 raft_consensus.cc:348] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.729723 469 raft_consensus.cc:374] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:31.730423 469 raft_consensus.cc:729] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Initialized, Role: FOLLOWER
I20260808 01:08:31.731626 469 consensus_queue.cc:260] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.734515 469 ts_tablet_manager.cc:1428] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260808 01:08:31.735678 375 heartbeater.cc:503] Master 127.31.212.126:39845 was elected leader, sending a full tablet report...
I20260808 01:08:31.735672 469 tablet_bootstrap.cc:492] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954: Bootstrap starting.
W20260808 01:08:31.738526 32762 tablet.cc:2369] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:08:31.739298 471 ts_tablet_manager.cc:1428] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent starting tablet: real 0.031s user 0.010s sys 0.005s
I20260808 01:08:31.742261 471 tablet_bootstrap.cc:492] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a: Bootstrap starting.
I20260808 01:08:31.747046 469 tablet_bootstrap.cc:654] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:31.747071 476 raft_consensus.cc:482] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:31.751173 476 raft_consensus.cc:504] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.754297 470 tablet_bootstrap.cc:654] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:31.772522 471 tablet_bootstrap.cc:654] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:31.776605 476 leader_election.cc:288] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:31.785920 470 tablet_bootstrap.cc:492] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4: No bootstrap required, opened a new log
I20260808 01:08:31.786592 470 ts_tablet_manager.cc:1397] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4: Time spent bootstrapping tablet: real 0.059s user 0.013s sys 0.005s
I20260808 01:08:31.792158 471 tablet_bootstrap.cc:492] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a: No bootstrap required, opened a new log
I20260808 01:08:31.795766 471 ts_tablet_manager.cc:1397] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.008s
I20260808 01:08:31.787428 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" is_pre_election: true
W20260808 01:08:31.798537 32696 leader_election.cc:341] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:08:31.793627 469 tablet_bootstrap.cc:492] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954: No bootstrap required, opened a new log
I20260808 01:08:31.798892 470 raft_consensus.cc:348] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.801298 470 raft_consensus.cc:374] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:31.799124 471 raft_consensus.cc:348] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.800889 469 ts_tablet_manager.cc:1397] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954: Time spent bootstrapping tablet: real 0.065s user 0.015s sys 0.004s
I20260808 01:08:31.802011 471 raft_consensus.cc:374] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:31.801688 470 raft_consensus.cc:729] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Initialized, Role: FOLLOWER
I20260808 01:08:31.802379 471 raft_consensus.cc:729] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Initialized, Role: FOLLOWER
I20260808 01:08:31.803352 470 consensus_queue.cc:260] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.803929 471 consensus_queue.cc:260] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.808974 471 ts_tablet_manager.cc:1428] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent starting tablet: real 0.013s user 0.004s sys 0.006s
I20260808 01:08:31.809983 471 tablet_bootstrap.cc:492] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a: Bootstrap starting.
I20260808 01:08:31.811235 470 ts_tablet_manager.cc:1428] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4: Time spent starting tablet: real 0.016s user 0.002s sys 0.005s
I20260808 01:08:31.812335 470 tablet_bootstrap.cc:492] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4: Bootstrap starting.
I20260808 01:08:31.817576 471 tablet_bootstrap.cc:654] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:31.805887 469 raft_consensus.cc:348] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.820024 469 raft_consensus.cc:374] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:31.820436 469 raft_consensus.cc:729] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Initialized, Role: FOLLOWER
I20260808 01:08:31.822140 469 consensus_queue.cc:260] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.824666 469 ts_tablet_manager.cc:1428] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954: Time spent starting tablet: real 0.022s user 0.007s sys 0.001s
I20260808 01:08:31.825836 469 tablet_bootstrap.cc:492] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954: Bootstrap starting.
I20260808 01:08:31.828198 425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954" is_pre_election: true
W20260808 01:08:31.829787 32696 leader_election.cc:341] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:08:31.830348 32696 leader_election.cc:302] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4; no voters: 90b6685012e44e47a191e25ffc352954, 97ca130eec6c4aa98ca7208cd35ed98a
I20260808 01:08:31.831372 470 tablet_bootstrap.cc:654] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:31.831732 476 raft_consensus.cc:2726] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:08:31.832857 469 tablet_bootstrap.cc:654] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:31.836839 474 raft_consensus.cc:482] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:31.837668 474 raft_consensus.cc:504] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.844534 474 leader_election.cc:288] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:31.846427 471 tablet_bootstrap.cc:492] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a: No bootstrap required, opened a new log
I20260808 01:08:31.847319 471 ts_tablet_manager.cc:1397] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent bootstrapping tablet: real 0.038s user 0.016s sys 0.005s
I20260808 01:08:31.850474 471 raft_consensus.cc:348] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.852002 471 raft_consensus.cc:374] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:31.851850 478 raft_consensus.cc:482] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:31.853087 471 raft_consensus.cc:729] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Initialized, Role: FOLLOWER
I20260808 01:08:31.852527 478 raft_consensus.cc:504] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:31.854072 471 consensus_queue.cc:260] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.857023 478 leader_election.cc:288] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:31.865334 474 raft_consensus.cc:482] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:31.866041 474 raft_consensus.cc:504] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.868638 474 leader_election.cc:288] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:31.870314 476 raft_consensus.cc:482] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:31.870929 476 raft_consensus.cc:504] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.873193 476 leader_election.cc:288] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:31.876389 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbc69178831403785892df61938ee97" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" is_pre_election: true
I20260808 01:08:31.877388 470 tablet_bootstrap.cc:492] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4: No bootstrap required, opened a new log
I20260808 01:08:31.877894 470 ts_tablet_manager.cc:1397] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4: Time spent bootstrapping tablet: real 0.066s user 0.006s sys 0.008s
I20260808 01:08:31.878628 425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbc69178831403785892df61938ee97" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954" is_pre_election: true
W20260808 01:08:31.881182 32696 leader_election.cc:341] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:08:31.892707 32696 leader_election.cc:341] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:08:31.896975 32696 leader_election.cc:302] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4; no voters: 90b6685012e44e47a191e25ffc352954, 97ca130eec6c4aa98ca7208cd35ed98a
I20260808 01:08:31.892755 471 ts_tablet_manager.cc:1428] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent starting tablet: real 0.045s user 0.004s sys 0.008s
I20260808 01:08:31.900658 476 raft_consensus.cc:2726] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:08:31.902197 469 tablet_bootstrap.cc:492] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954: No bootstrap required, opened a new log
I20260808 01:08:31.902590 469 ts_tablet_manager.cc:1397] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954: Time spent bootstrapping tablet: real 0.077s user 0.014s sys 0.000s
I20260808 01:08:31.906050 469 raft_consensus.cc:348] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:31.898233 470 raft_consensus.cc:348] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:31.908228 469 raft_consensus.cc:374] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:31.911355 469 raft_consensus.cc:729] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Initialized, Role: FOLLOWER
I20260808 01:08:31.909425 470 raft_consensus.cc:374] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:31.912015 470 raft_consensus.cc:729] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Initialized, Role: FOLLOWER
I20260808 01:08:31.911681 471 tablet_bootstrap.cc:492] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a: Bootstrap starting.
I20260808 01:08:31.912499 469 consensus_queue.cc:260] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:31.916056 490 raft_consensus.cc:482] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:31.916671 490 raft_consensus.cc:504] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.920650 490 leader_election.cc:288] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:31.921311 470 consensus_queue.cc:260] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:31.920939 469 ts_tablet_manager.cc:1428] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954: Time spent starting tablet: real 0.018s user 0.007s sys 0.002s
I20260808 01:08:31.927166 469 tablet_bootstrap.cc:492] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954: Bootstrap starting.
I20260808 01:08:31.930169 470 ts_tablet_manager.cc:1428] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4: Time spent starting tablet: real 0.052s user 0.007s sys 0.005s
I20260808 01:08:31.933032 470 tablet_bootstrap.cc:492] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4: Bootstrap starting.
I20260808 01:08:31.941936 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4" is_pre_election: true
I20260808 01:08:31.940959 32734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8a848bc0b4438b8ae3e5a5ff59f3b6" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4" is_pre_election: true
W20260808 01:08:31.948439 387 leader_election.cc:341] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:08:31.949735 471 tablet_bootstrap.cc:654] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a: Neither blocks nor log segments found. Creating new log.
W20260808 01:08:31.949303 387 leader_election.cc:341] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:08:31.955243 470 tablet_bootstrap.cc:654] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:31.956647 469 tablet_bootstrap.cc:654] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:31.973320 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" is_pre_election: true
I20260808 01:08:31.973943 347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8a848bc0b4438b8ae3e5a5ff59f3b6" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" is_pre_election: true
I20260808 01:08:31.974310 348 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 0.
I20260808 01:08:31.976279 32734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4" is_pre_election: true
W20260808 01:08:31.978652 307 leader_election.cc:341] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:08:31.979965 425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e59a231e74d54b4859d7f52e8f0d1" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954" is_pre_election: true
I20260808 01:08:31.979985 424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954" is_pre_election: true
I20260808 01:08:31.981029 424 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 0.
I20260808 01:08:31.975528 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e59a231e74d54b4859d7f52e8f0d1" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4" is_pre_election: true
I20260808 01:08:31.982101 32735 raft_consensus.cc:2454] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 0.
W20260808 01:08:31.982520 387 leader_election.cc:341] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:08:31.982573 307 leader_election.cc:302] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90b6685012e44e47a191e25ffc352954, 97ca130eec6c4aa98ca7208cd35ed98a; no voters: 15218c77abfe47d9ab4765853cc3b2c4
I20260808 01:08:31.981047 425 raft_consensus.cc:2454] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 0.
I20260808 01:08:31.983079 387 leader_election.cc:302] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90b6685012e44e47a191e25ffc352954; no voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a
I20260808 01:08:31.983570 490 raft_consensus.cc:2781] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:08:31.983865 474 raft_consensus.cc:2726] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:08:31.983986 490 raft_consensus.cc:482] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:08:31.984009 307 leader_election.cc:302] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:31.984438 490 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:31.984903 478 raft_consensus.cc:2781] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:08:31.985280 478 raft_consensus.cc:482] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:08:31.985634 478 raft_consensus.cc:3037] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:31.986677 471 tablet_bootstrap.cc:492] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a: No bootstrap required, opened a new log
I20260808 01:08:31.987185 471 ts_tablet_manager.cc:1397] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent bootstrapping tablet: real 0.076s user 0.017s sys 0.011s
I20260808 01:08:31.990191 470 tablet_bootstrap.cc:492] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4: No bootstrap required, opened a new log
I20260808 01:08:31.990224 490 raft_consensus.cc:504] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.990792 470 ts_tablet_manager.cc:1397] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4: Time spent bootstrapping tablet: real 0.059s user 0.015s sys 0.003s
I20260808 01:08:31.984606 387 leader_election.cc:302] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90b6685012e44e47a191e25ffc352954, 97ca130eec6c4aa98ca7208cd35ed98a; no voters: 15218c77abfe47d9ab4765853cc3b2c4
I20260808 01:08:31.992619 490 leader_election.cc:288] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:31.993114 474 raft_consensus.cc:2781] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:08:31.993484 474 raft_consensus.cc:482] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:08:31.993801 474 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:31.994100 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:31.994604 470 raft_consensus.cc:348] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.995231 470 raft_consensus.cc:374] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:31.995414 470 raft_consensus.cc:729] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Initialized, Role: FOLLOWER
W20260808 01:08:31.995635 307 leader_election.cc:341] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:08:31.996297 471 raft_consensus.cc:348] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:31.997269 469 tablet_bootstrap.cc:492] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954: No bootstrap required, opened a new log
I20260808 01:08:31.997222 471 raft_consensus.cc:374] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:31.997745 469 ts_tablet_manager.cc:1397] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954: Time spent bootstrapping tablet: real 0.071s user 0.012s sys 0.009s
I20260808 01:08:31.997799 471 raft_consensus.cc:729] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Initialized, Role: FOLLOWER
I20260808 01:08:31.998597 471 consensus_queue.cc:260] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.000890 471 ts_tablet_manager.cc:1428] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent starting tablet: real 0.013s user 0.004s sys 0.006s
I20260808 01:08:32.000578 469 raft_consensus.cc:348] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.001328 469 raft_consensus.cc:374] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:32.001664 469 raft_consensus.cc:729] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Initialized, Role: FOLLOWER
I20260808 01:08:32.001798 471 tablet_bootstrap.cc:492] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a: Bootstrap starting.
I20260808 01:08:32.002672 425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:32.003625 470 consensus_queue.cc:260] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.009836 471 tablet_bootstrap.cc:654] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:32.004827 474 raft_consensus.cc:504] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.007202 469 consensus_queue.cc:260] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.016428 425 raft_consensus.cc:2379] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 97ca130eec6c4aa98ca7208cd35ed98a in current term 1: Already voted for candidate 90b6685012e44e47a191e25ffc352954 in this term.
I20260808 01:08:32.017302 474 leader_election.cc:288] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:32.014830 478 raft_consensus.cc:504] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:32.018266 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:32.019544 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:32.020025 478 leader_election.cc:288] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:32.020439 348 raft_consensus.cc:2379] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 90b6685012e44e47a191e25ffc352954 in current term 1: Already voted for candidate 97ca130eec6c4aa98ca7208cd35ed98a in this term.
I20260808 01:08:32.006888 470 ts_tablet_manager.cc:1428] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260808 01:08:32.020725 307 leader_election.cc:302] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97ca130eec6c4aa98ca7208cd35ed98a; no voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954
I20260808 01:08:32.021996 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e59a231e74d54b4859d7f52e8f0d1" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
W20260808 01:08:32.022394 387 leader_election.cc:341] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:08:32.022851 32735 raft_consensus.cc:3037] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.024050 387 leader_election.cc:302] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90b6685012e44e47a191e25ffc352954; no voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a
I20260808 01:08:32.025491 425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e59a231e74d54b4859d7f52e8f0d1" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:32.025965 478 raft_consensus.cc:2726] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:08:32.026113 425 raft_consensus.cc:3037] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.027089 474 raft_consensus.cc:2726] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:08:32.027860 474 raft_consensus.cc:482] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:32.028431 474 raft_consensus.cc:504] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.031158 474 leader_election.cc:288] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:32.033265 32734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f5c989c2b7409c8ee0112f26936deb" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4" is_pre_election: true
I20260808 01:08:32.033994 32734 raft_consensus.cc:2454] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 0.
I20260808 01:08:32.036125 470 tablet_bootstrap.cc:492] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4: Bootstrap starting.
I20260808 01:08:32.035773 387 leader_election.cc:302] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954; no voters:
I20260808 01:08:32.037329 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f5c989c2b7409c8ee0112f26936deb" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" is_pre_election: true
I20260808 01:08:32.039433 32735 raft_consensus.cc:2454] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 1.
I20260808 01:08:32.040777 307 leader_election.cc:302] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
W20260808 01:08:32.041515 387 leader_election.cc:341] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:08:32.041630 478 raft_consensus.cc:2781] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:08:32.039538 425 raft_consensus.cc:2454] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 1.
I20260808 01:08:32.042511 478 raft_consensus.cc:686] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Becoming Leader. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:32.045858 478 consensus_queue.cc:237] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:32.044283 474 raft_consensus.cc:2781] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:08:32.053901 474 raft_consensus.cc:482] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:08:32.056152 474 raft_consensus.cc:3037] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.056414 469 ts_tablet_manager.cc:1428] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954: Time spent starting tablet: real 0.058s user 0.022s sys 0.019s
I20260808 01:08:32.057675 469 tablet_bootstrap.cc:492] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954: Bootstrap starting.
I20260808 01:08:32.058054 470 tablet_bootstrap.cc:654] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:32.068988 469 tablet_bootstrap.cc:654] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:32.072903 471 tablet_bootstrap.cc:492] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a: No bootstrap required, opened a new log
I20260808 01:08:32.073563 471 ts_tablet_manager.cc:1397] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent bootstrapping tablet: real 0.072s user 0.018s sys 0.012s
I20260808 01:08:32.074468 499 raft_consensus.cc:482] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:32.075102 499 raft_consensus.cc:504] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.078961 474 raft_consensus.cc:504] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.079881 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9ecb475d7143bd94ed0590fd4c5368" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4" is_pre_election: true
I20260808 01:08:32.081092 474 leader_election.cc:288] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:32.081496 499 leader_election.cc:288] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:32.081589 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9ecb475d7143bd94ed0590fd4c5368" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" is_pre_election: true
W20260808 01:08:32.082810 387 leader_election.cc:341] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:08:32.083501 387 leader_election.cc:341] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:08:32.083928 387 leader_election.cc:302] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90b6685012e44e47a191e25ffc352954; no voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a
I20260808 01:08:32.084077 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f5c989c2b7409c8ee0112f26936deb" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:32.084450 471 raft_consensus.cc:348] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
W20260808 01:08:32.085207 387 leader_election.cc:341] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:08:32.085423 471 raft_consensus.cc:374] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:32.085457 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f5c989c2b7409c8ee0112f26936deb" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:32.086154 32735 raft_consensus.cc:3037] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.086524 499 raft_consensus.cc:2726] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:08:32.085995 471 raft_consensus.cc:729] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Initialized, Role: FOLLOWER
I20260808 01:08:32.092734 471 consensus_queue.cc:260] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.096439 470 tablet_bootstrap.cc:492] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4: No bootstrap required, opened a new log
I20260808 01:08:32.097086 470 ts_tablet_manager.cc:1397] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4: Time spent bootstrapping tablet: real 0.061s user 0.015s sys 0.004s
I20260808 01:08:32.097273 469 tablet_bootstrap.cc:492] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954: No bootstrap required, opened a new log
I20260808 01:08:32.098018 469 ts_tablet_manager.cc:1397] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954: Time spent bootstrapping tablet: real 0.041s user 0.018s sys 0.003s
I20260808 01:08:32.099125 32735 raft_consensus.cc:2454] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 1.
I20260808 01:08:32.100332 387 leader_election.cc:302] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954; no voters: 97ca130eec6c4aa98ca7208cd35ed98a
I20260808 01:08:32.101161 499 raft_consensus.cc:2781] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:08:32.101702 499 raft_consensus.cc:686] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 1 LEADER]: Becoming Leader. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Running, Role: LEADER
I20260808 01:08:32.102329 499 consensus_queue.cc:237] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.101214 470 raft_consensus.cc:348] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.103150 470 raft_consensus.cc:374] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:32.104353 471 ts_tablet_manager.cc:1428] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent starting tablet: real 0.030s user 0.007s sys 0.002s
I20260808 01:08:32.105337 470 raft_consensus.cc:729] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Initialized, Role: FOLLOWER
I20260808 01:08:32.103492 32625 catalog_manager.cc:5495] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a reported cstate change: term changed from 0 to 1, leader changed from <none> to 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66). New cstate: current_term: 1 leader_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: HEALTHY } } }
I20260808 01:08:32.106508 470 consensus_queue.cc:260] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.108073 469 raft_consensus.cc:348] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.109694 471 tablet_bootstrap.cc:492] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a: Bootstrap starting.
I20260808 01:08:32.110260 470 ts_tablet_manager.cc:1428] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20260808 01:08:32.111011 470 tablet_bootstrap.cc:492] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4: Bootstrap starting.
I20260808 01:08:32.109033 469 raft_consensus.cc:374] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:32.112182 469 raft_consensus.cc:729] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Initialized, Role: FOLLOWER
I20260808 01:08:32.112990 469 consensus_queue.cc:260] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.115620 469 ts_tablet_manager.cc:1428] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20260808 01:08:32.116649 469 tablet_bootstrap.cc:492] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954: Bootstrap starting.
I20260808 01:08:32.117272 471 tablet_bootstrap.cc:654] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:32.117339 470 tablet_bootstrap.cc:654] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4: Neither blocks nor log segments found. Creating new log.
W20260808 01:08:32.129454 32678 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:08:32.137279 469 tablet_bootstrap.cc:654] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:32.144164 478 raft_consensus.cc:482] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:32.144810 478 raft_consensus.cc:504] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.148059 478 leader_election.cc:288] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:32.148748 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da895fe9b014f5d8f2acede20d67c53" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4" is_pre_election: true
I20260808 01:08:32.149554 32735 raft_consensus.cc:2454] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 0.
I20260808 01:08:32.151104 307 leader_election.cc:302] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:32.151889 425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da895fe9b014f5d8f2acede20d67c53" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954" is_pre_election: true
I20260808 01:08:32.152495 478 raft_consensus.cc:2781] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:08:32.152978 478 raft_consensus.cc:482] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:08:32.153398 307 leader_election.cc:341] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:08:32.153505 478 raft_consensus.cc:3037] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.164052 471 tablet_bootstrap.cc:492] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a: No bootstrap required, opened a new log
I20260808 01:08:32.164839 471 ts_tablet_manager.cc:1397] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent bootstrapping tablet: real 0.055s user 0.017s sys 0.005s
I20260808 01:08:32.167940 470 tablet_bootstrap.cc:492] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4: No bootstrap required, opened a new log
I20260808 01:08:32.168740 470 ts_tablet_manager.cc:1397] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4: Time spent bootstrapping tablet: real 0.058s user 0.007s sys 0.015s
I20260808 01:08:32.169941 469 tablet_bootstrap.cc:492] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954: No bootstrap required, opened a new log
I20260808 01:08:32.170491 471 raft_consensus.cc:348] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.170531 469 ts_tablet_manager.cc:1397] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954: Time spent bootstrapping tablet: real 0.054s user 0.023s sys 0.006s
I20260808 01:08:32.173380 478 raft_consensus.cc:504] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.177279 469 raft_consensus.cc:348] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.177956 478 leader_election.cc:288] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:32.173413 470 raft_consensus.cc:348] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.178391 469 raft_consensus.cc:374] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:32.178627 470 raft_consensus.cc:374] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:32.178987 469 raft_consensus.cc:729] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Initialized, Role: FOLLOWER
I20260808 01:08:32.178992 470 raft_consensus.cc:729] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Initialized, Role: FOLLOWER
I20260808 01:08:32.171308 471 raft_consensus.cc:374] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:32.180294 471 raft_consensus.cc:729] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Initialized, Role: FOLLOWER
I20260808 01:08:32.180208 425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da895fe9b014f5d8f2acede20d67c53" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:32.180228 470 consensus_queue.cc:260] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.181085 469 consensus_queue.cc:260] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.181845 471 consensus_queue.cc:260] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
W20260808 01:08:32.182819 307 leader_election.cc:341] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:08:32.179750 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da895fe9b014f5d8f2acede20d67c53" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:32.183545 32735 raft_consensus.cc:3037] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.184705 471 ts_tablet_manager.cc:1428] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent starting tablet: real 0.020s user 0.006s sys 0.003s
I20260808 01:08:32.185817 471 tablet_bootstrap.cc:492] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a: Bootstrap starting.
I20260808 01:08:32.186491 470 ts_tablet_manager.cc:1428] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4: Time spent starting tablet: real 0.017s user 0.005s sys 0.005s
I20260808 01:08:32.186781 469 ts_tablet_manager.cc:1428] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954: Time spent starting tablet: real 0.015s user 0.006s sys 0.003s
I20260808 01:08:32.187505 470 tablet_bootstrap.cc:492] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4: Bootstrap starting.
I20260808 01:08:32.188226 469 tablet_bootstrap.cc:492] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954: Bootstrap starting.
I20260808 01:08:32.190842 32735 raft_consensus.cc:2454] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 1.
I20260808 01:08:32.192302 307 leader_election.cc:302] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a; no voters: 90b6685012e44e47a191e25ffc352954
I20260808 01:08:32.193442 478 raft_consensus.cc:2781] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:08:32.194300 478 raft_consensus.cc:686] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Becoming Leader. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:32.198854 470 tablet_bootstrap.cc:654] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:32.198866 469 tablet_bootstrap.cc:654] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:32.201481 471 tablet_bootstrap.cc:654] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:32.195464 478 consensus_queue.cc:237] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.207821 470 tablet_bootstrap.cc:492] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4: No bootstrap required, opened a new log
I20260808 01:08:32.208935 470 ts_tablet_manager.cc:1397] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.006s
I20260808 01:08:32.217976 470 raft_consensus.cc:348] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.218839 470 raft_consensus.cc:374] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:32.219138 470 raft_consensus.cc:729] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Initialized, Role: FOLLOWER
I20260808 01:08:32.219921 470 consensus_queue.cc:260] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.222606 470 ts_tablet_manager.cc:1428] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20260808 01:08:32.223497 478 raft_consensus.cc:482] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:32.224124 470 tablet_bootstrap.cc:492] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 15218c77abfe47d9ab4765853cc3b2c4: Bootstrap starting.
I20260808 01:08:32.224053 478 raft_consensus.cc:504] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.221642 32627 catalog_manager.cc:5495] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90b6685012e44e47a191e25ffc352954 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "90b6685012e44e47a191e25ffc352954" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: HEALTHY } } }
I20260808 01:08:32.226714 478 leader_election.cc:288] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:32.228212 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a4791d89904ee29ff78ef43c021b65" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4" is_pre_election: true
I20260808 01:08:32.228950 32735 raft_consensus.cc:2454] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 0.
I20260808 01:08:32.233992 425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a4791d89904ee29ff78ef43c021b65" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954" is_pre_election: true
I20260808 01:08:32.234714 425 raft_consensus.cc:2454] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 0.
I20260808 01:08:32.234791 307 leader_election.cc:302] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:32.235673 478 raft_consensus.cc:2781] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:08:32.236258 478 raft_consensus.cc:482] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:08:32.236740 478 raft_consensus.cc:3037] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.250514 478 raft_consensus.cc:504] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.258266 470 tablet_bootstrap.cc:654] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 15218c77abfe47d9ab4765853cc3b2c4: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:32.259646 478 leader_election.cc:288] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:32.261365 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a4791d89904ee29ff78ef43c021b65" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:32.262038 32735 raft_consensus.cc:3037] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.264935 469 tablet_bootstrap.cc:492] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954: No bootstrap required, opened a new log
I20260808 01:08:32.265657 469 ts_tablet_manager.cc:1397] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954: Time spent bootstrapping tablet: real 0.078s user 0.010s sys 0.046s
I20260808 01:08:32.266064 425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a4791d89904ee29ff78ef43c021b65" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:32.266999 425 raft_consensus.cc:3037] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.270182 469 raft_consensus.cc:348] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.271009 469 raft_consensus.cc:374] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:32.271359 469 raft_consensus.cc:729] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Initialized, Role: FOLLOWER
I20260808 01:08:32.272068 469 consensus_queue.cc:260] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.272940 471 tablet_bootstrap.cc:492] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a: No bootstrap required, opened a new log
I20260808 01:08:32.273561 471 ts_tablet_manager.cc:1397] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent bootstrapping tablet: real 0.088s user 0.006s sys 0.016s
I20260808 01:08:32.274602 469 ts_tablet_manager.cc:1428] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260808 01:08:32.275624 469 tablet_bootstrap.cc:492] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954: Bootstrap starting.
I20260808 01:08:32.280608 32735 raft_consensus.cc:2454] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 1.
I20260808 01:08:32.282200 469 tablet_bootstrap.cc:654] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:32.277889 471 raft_consensus.cc:348] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.283640 307 leader_election.cc:302] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:32.285061 471 raft_consensus.cc:374] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:32.287330 478 raft_consensus.cc:2781] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:08:32.289949 478 raft_consensus.cc:686] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Becoming Leader. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:32.291330 478 consensus_queue.cc:237] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.287670 471 raft_consensus.cc:729] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Initialized, Role: FOLLOWER
I20260808 01:08:32.295347 425 raft_consensus.cc:2454] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 1.
I20260808 01:08:32.302968 470 tablet_bootstrap.cc:492] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 15218c77abfe47d9ab4765853cc3b2c4: No bootstrap required, opened a new log
I20260808 01:08:32.303565 470 ts_tablet_manager.cc:1397] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 15218c77abfe47d9ab4765853cc3b2c4: Time spent bootstrapping tablet: real 0.080s user 0.014s sys 0.003s
I20260808 01:08:32.302384 471 consensus_queue.cc:260] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.306375 471 ts_tablet_manager.cc:1428] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent starting tablet: real 0.032s user 0.007s sys 0.002s
I20260808 01:08:32.305596 32625 catalog_manager.cc:5495] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a reported cstate change: term changed from 0 to 1, leader changed from <none> to 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66). New cstate: current_term: 1 leader_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } }
I20260808 01:08:32.306964 470 raft_consensus.cc:348] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.308174 470 raft_consensus.cc:374] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:32.308550 470 raft_consensus.cc:729] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Initialized, Role: FOLLOWER
I20260808 01:08:32.309329 470 consensus_queue.cc:260] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 15218c77abfe47d9ab4765853cc3b2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.310531 469 tablet_bootstrap.cc:492] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954: No bootstrap required, opened a new log
I20260808 01:08:32.310988 469 ts_tablet_manager.cc:1397] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.011s
I20260808 01:08:32.312295 470 ts_tablet_manager.cc:1428] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 15218c77abfe47d9ab4765853cc3b2c4: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20260808 01:08:32.313345 470 tablet_bootstrap.cc:492] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4: Bootstrap starting.
I20260808 01:08:32.313982 471 tablet_bootstrap.cc:492] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a: Bootstrap starting.
I20260808 01:08:32.313704 469 raft_consensus.cc:348] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.314512 469 raft_consensus.cc:374] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:32.314925 469 raft_consensus.cc:729] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Initialized, Role: FOLLOWER
I20260808 01:08:32.315690 469 consensus_queue.cc:260] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.320377 469 ts_tablet_manager.cc:1428] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260808 01:08:32.321326 470 tablet_bootstrap.cc:654] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:32.322347 471 tablet_bootstrap.cc:654] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:32.332890 499 raft_consensus.cc:482] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:32.333673 499 raft_consensus.cc:504] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.337905 470 tablet_bootstrap.cc:492] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4: No bootstrap required, opened a new log
I20260808 01:08:32.338500 470 ts_tablet_manager.cc:1397] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4: Time spent bootstrapping tablet: real 0.025s user 0.017s sys 0.004s
I20260808 01:08:32.341142 470 raft_consensus.cc:348] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.341873 470 raft_consensus.cc:374] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:32.342154 470 raft_consensus.cc:729] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Initialized, Role: FOLLOWER
I20260808 01:08:32.342805 470 consensus_queue.cc:260] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.345590 470 ts_tablet_manager.cc:1428] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4: Time spent starting tablet: real 0.007s user 0.000s sys 0.008s
I20260808 01:08:32.347661 471 tablet_bootstrap.cc:492] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a: No bootstrap required, opened a new log
I20260808 01:08:32.348289 471 ts_tablet_manager.cc:1397] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.004s
I20260808 01:08:32.351155 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbc69178831403785892df61938ee97" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4" is_pre_election: true
I20260808 01:08:32.350245 499 leader_election.cc:288] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:32.351393 471 raft_consensus.cc:348] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.352236 32735 raft_consensus.cc:2454] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 0.
I20260808 01:08:32.352353 471 raft_consensus.cc:374] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:32.352866 471 raft_consensus.cc:729] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Initialized, Role: FOLLOWER
I20260808 01:08:32.353487 478 raft_consensus.cc:482] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:32.353607 471 consensus_queue.cc:260] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.354147 478 raft_consensus.cc:504] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.353971 32626 catalog_manager.cc:5495] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a reported cstate change: term changed from 0 to 1, leader changed from <none> to 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66). New cstate: current_term: 1 leader_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } }
I20260808 01:08:32.357288 471 ts_tablet_manager.cc:1428] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260808 01:08:32.357759 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8a848bc0b4438b8ae3e5a5ff59f3b6" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4" is_pre_election: true
I20260808 01:08:32.358384 471 tablet_bootstrap.cc:492] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a: Bootstrap starting.
I20260808 01:08:32.358505 32735 raft_consensus.cc:2454] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 0.
I20260808 01:08:32.359783 307 leader_election.cc:302] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:32.360668 490 raft_consensus.cc:2781] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:08:32.361032 490 raft_consensus.cc:482] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:08:32.361377 490 raft_consensus.cc:3037] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.361577 478 leader_election.cc:288] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:32.363329 387 leader_election.cc:302] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954; no voters:
I20260808 01:08:32.363421 425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8a848bc0b4438b8ae3e5a5ff59f3b6" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954" is_pre_election: true
I20260808 01:08:32.364125 425 raft_consensus.cc:2454] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 0.
I20260808 01:08:32.364276 499 raft_consensus.cc:2781] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:08:32.364655 499 raft_consensus.cc:482] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:08:32.364989 499 raft_consensus.cc:3037] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.364953 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbc69178831403785892df61938ee97" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" is_pre_election: true
I20260808 01:08:32.365768 348 raft_consensus.cc:2454] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 0.
I20260808 01:08:32.371987 490 raft_consensus.cc:504] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.372489 499 raft_consensus.cc:504] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.374591 499 leader_election.cc:288] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:32.376205 32734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbc69178831403785892df61938ee97" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:32.376890 32734 raft_consensus.cc:3037] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.377282 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbc69178831403785892df61938ee97" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:32.377885 348 raft_consensus.cc:3037] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.380728 471 tablet_bootstrap.cc:654] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a: Neither blocks nor log segments found. Creating new log.
I20260808 01:08:32.381460 490 leader_election.cc:288] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:32.381662 425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8a848bc0b4438b8ae3e5a5ff59f3b6" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:32.382349 425 raft_consensus.cc:3037] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.383870 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8a848bc0b4438b8ae3e5a5ff59f3b6" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:32.387727 348 raft_consensus.cc:2454] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 1.
I20260808 01:08:32.388031 32735 raft_consensus.cc:3037] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.389072 387 leader_election.cc:302] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90b6685012e44e47a191e25ffc352954, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:32.392058 499 raft_consensus.cc:2781] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:08:32.387727 32734 raft_consensus.cc:2454] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 1.
I20260808 01:08:32.392861 499 raft_consensus.cc:686] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 1 LEADER]: Becoming Leader. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Running, Role: LEADER
I20260808 01:08:32.391813 425 raft_consensus.cc:2454] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 1.
I20260808 01:08:32.395109 307 leader_election.cc:302] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90b6685012e44e47a191e25ffc352954, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:32.396629 490 raft_consensus.cc:2781] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:08:32.397380 490 raft_consensus.cc:686] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Becoming Leader. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:32.398419 490 consensus_queue.cc:237] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.399880 32735 raft_consensus.cc:2454] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 1.
I20260808 01:08:32.395995 499 consensus_queue.cc:237] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.402823 471 tablet_bootstrap.cc:492] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a: No bootstrap required, opened a new log
I20260808 01:08:32.403282 471 ts_tablet_manager.cc:1397] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent bootstrapping tablet: real 0.045s user 0.017s sys 0.005s
I20260808 01:08:32.406375 471 raft_consensus.cc:348] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.407142 471 raft_consensus.cc:374] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:08:32.407440 471 raft_consensus.cc:729] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Initialized, Role: FOLLOWER
I20260808 01:08:32.408092 471 consensus_queue.cc:260] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.410454 471 ts_tablet_manager.cc:1428] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260808 01:08:32.421891 32626 catalog_manager.cc:5495] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90b6685012e44e47a191e25ffc352954 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "90b6685012e44e47a191e25ffc352954" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: HEALTHY } } }
I20260808 01:08:32.423000 32625 catalog_manager.cc:5495] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a reported cstate change: term changed from 0 to 1, leader changed from <none> to 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66). New cstate: current_term: 1 leader_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } }
I20260808 01:08:32.470669 499 raft_consensus.cc:482] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:32.471349 499 raft_consensus.cc:504] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.473747 499 leader_election.cc:288] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:32.474606 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4" is_pre_election: true
I20260808 01:08:32.474874 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" is_pre_election: true
I20260808 01:08:32.475253 32735 raft_consensus.cc:2454] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 0.
I20260808 01:08:32.475708 348 raft_consensus.cc:2454] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 0.
I20260808 01:08:32.476686 387 leader_election.cc:302] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954; no voters:
I20260808 01:08:32.477526 499 raft_consensus.cc:2781] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:08:32.477936 499 raft_consensus.cc:482] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:08:32.478325 499 raft_consensus.cc:3037] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.486260 499 raft_consensus.cc:504] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.488646 499 leader_election.cc:288] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:32.489663 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:32.490228 32735 raft_consensus.cc:3037] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.490470 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:32.491061 348 raft_consensus.cc:3037] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.497419 32735 raft_consensus.cc:2454] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 1.
I20260808 01:08:32.497443 348 raft_consensus.cc:2454] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 1.
I20260808 01:08:32.498693 387 leader_election.cc:302] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90b6685012e44e47a191e25ffc352954, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:32.499516 499 raft_consensus.cc:2781] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:08:32.500150 499 raft_consensus.cc:686] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 1 LEADER]: Becoming Leader. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Running, Role: LEADER
I20260808 01:08:32.501062 474 raft_consensus.cc:482] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:32.501588 474 raft_consensus.cc:504] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.501075 499 consensus_queue.cc:237] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.503681 474 leader_election.cc:288] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:32.505015 32735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da895fe9b014f5d8f2acede20d67c53" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4" is_pre_election: true
I20260808 01:08:32.506018 32735 raft_consensus.cc:2379] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90b6685012e44e47a191e25ffc352954 in current term 1: Already voted for candidate 97ca130eec6c4aa98ca7208cd35ed98a in this term.
I20260808 01:08:32.508059 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da895fe9b014f5d8f2acede20d67c53" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" is_pre_election: true
I20260808 01:08:32.510143 387 leader_election.cc:302] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90b6685012e44e47a191e25ffc352954; no voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a
I20260808 01:08:32.511377 499 raft_consensus.cc:3037] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.512449 32625 catalog_manager.cc:5495] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90b6685012e44e47a191e25ffc352954 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "90b6685012e44e47a191e25ffc352954" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: HEALTHY } } }
I20260808 01:08:32.516913 499 raft_consensus.cc:2726] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:08:32.568352 476 raft_consensus.cc:482] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:32.569047 476 raft_consensus.cc:504] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.571216 476 leader_election.cc:288] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:32.572412 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" is_pre_election: true
I20260808 01:08:32.572558 425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954" is_pre_election: true
I20260808 01:08:32.573256 425 raft_consensus.cc:2379] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 15218c77abfe47d9ab4765853cc3b2c4 in current term 1: Already voted for candidate 90b6685012e44e47a191e25ffc352954 in this term.
I20260808 01:08:32.573268 348 raft_consensus.cc:2379] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 15218c77abfe47d9ab4765853cc3b2c4 in current term 1: Already voted for candidate 97ca130eec6c4aa98ca7208cd35ed98a in this term.
I20260808 01:08:32.574577 32696 leader_election.cc:302] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4; no voters: 90b6685012e44e47a191e25ffc352954, 97ca130eec6c4aa98ca7208cd35ed98a
I20260808 01:08:32.575474 476 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.581136 476 raft_consensus.cc:2726] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:08:32.613083 490 consensus_queue.cc:1035] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:08:32.625771 499 consensus_queue.cc:1035] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:08:32.633015 490 consensus_queue.cc:1035] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:08:32.659785 533 raft_consensus.cc:482] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:32.660495 533 raft_consensus.cc:504] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.667371 348 raft_consensus.cc:3037] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.669448 533 leader_election.cc:288] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:32.671335 425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9ecb475d7143bd94ed0590fd4c5368" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954" is_pre_election: true
I20260808 01:08:32.671952 32734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9ecb475d7143bd94ed0590fd4c5368" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4" is_pre_election: true
I20260808 01:08:32.672181 425 raft_consensus.cc:2454] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 0.
I20260808 01:08:32.672744 32734 raft_consensus.cc:2454] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 0.
I20260808 01:08:32.673700 307 leader_election.cc:302] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90b6685012e44e47a191e25ffc352954, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:32.675258 533 raft_consensus.cc:2781] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:08:32.675686 533 raft_consensus.cc:482] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:08:32.676133 533 raft_consensus.cc:3037] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.679673 474 consensus_queue.cc:1035] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:08:32.684289 533 raft_consensus.cc:504] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.687076 32734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9ecb475d7143bd94ed0590fd4c5368" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:32.687808 32734 raft_consensus.cc:3037] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.689309 533 leader_election.cc:288] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:32.693082 425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9ecb475d7143bd94ed0590fd4c5368" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:32.693810 425 raft_consensus.cc:3037] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:08:32.695961 32734 raft_consensus.cc:2454] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 1.
I20260808 01:08:32.697665 307 leader_election.cc:302] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:32.699028 490 raft_consensus.cc:2781] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:08:32.702984 425 raft_consensus.cc:2454] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 1.
I20260808 01:08:32.697659 533 consensus_queue.cc:1035] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260808 01:08:32.707643 478 consensus_queue.cc:1035] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:08:32.710124 490 raft_consensus.cc:686] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Becoming Leader. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:32.711257 490 consensus_queue.cc:237] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:32.733879 32625 catalog_manager.cc:5495] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a reported cstate change: term changed from 0 to 1, leader changed from <none> to 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66). New cstate: current_term: 1 leader_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } }
I20260808 01:08:32.744386 478 consensus_queue.cc:1035] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:08:32.759994 533 consensus_queue.cc:1035] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:08:32.826575 474 consensus_queue.cc:1035] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:08:32.841979 499 consensus_queue.cc:1035] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:08:32.866020 490 consensus_queue.cc:1035] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:08:32.880055 490 consensus_queue.cc:1035] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:08:32.954802 528 consensus_queue.cc:1035] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:08:32.971925 528 consensus_queue.cc:1035] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:08:33.123914 548 consensus_queue.cc:1035] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:08:33.144450 478 consensus_queue.cc:1035] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 01:08:33.146526 32678 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260808 01:08:34.245599 32678 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:08:34.448045 556 raft_consensus.cc:482] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:34.448807 556 raft_consensus.cc:504] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:34.451922 556 leader_election.cc:288] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:34.474050 557 raft_consensus.cc:482] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:34.474666 557 raft_consensus.cc:504] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:34.475998 561 raft_consensus.cc:482] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:34.477192 557 leader_election.cc:288] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:34.479322 557 raft_consensus.cc:482] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90b6685012e44e47a191e25ffc352954)
I20260808 01:08:34.481421 557 raft_consensus.cc:504] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:34.483645 32730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4" is_pre_election: true
I20260808 01:08:34.476485 561 raft_consensus.cc:504] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:34.486203 32730 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 1.
I20260808 01:08:34.487519 32732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4" is_pre_election: true
I20260808 01:08:34.487871 561 leader_election.cc:288] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:34.491180 345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" is_pre_election: true
I20260808 01:08:34.493525 421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954" is_pre_election: true
I20260808 01:08:34.498139 345 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15218c77abfe47d9ab4765853cc3b2c4 in term 1.
I20260808 01:08:34.498245 421 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15218c77abfe47d9ab4765853cc3b2c4 in term 1.
I20260808 01:08:34.497448 557 leader_election.cc:288] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:34.499536 32696 leader_election.cc:302] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954; no voters:
I20260808 01:08:34.500855 561 raft_consensus.cc:2781] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:08:34.501248 561 raft_consensus.cc:482] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:08:34.501571 561 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:34.503392 421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954" is_pre_election: true
I20260808 01:08:34.504129 421 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 1.
I20260808 01:08:34.506716 421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954" is_pre_election: true
I20260808 01:08:34.507092 32731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4" is_pre_election: true
I20260808 01:08:34.509407 387 leader_election.cc:302] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954; no voters:
I20260808 01:08:34.510809 556 raft_consensus.cc:2781] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:08:34.511209 556 raft_consensus.cc:482] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:08:34.510990 307 leader_election.cc:302] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90b6685012e44e47a191e25ffc352954, 97ca130eec6c4aa98ca7208cd35ed98a; no voters: 15218c77abfe47d9ab4765853cc3b2c4
I20260808 01:08:34.511619 556 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:34.512285 346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" is_pre_election: true
I20260808 01:08:34.512982 346 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 1.
I20260808 01:08:34.512868 307 leader_election.cc:302] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97ca130eec6c4aa98ca7208cd35ed98a; no voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954
I20260808 01:08:34.513859 560 raft_consensus.cc:2726] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:08:34.513517 562 raft_consensus.cc:2781] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:08:34.514732 562 raft_consensus.cc:482] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:08:34.515069 562 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:34.509312 561 raft_consensus.cc:504] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:34.519529 561 leader_election.cc:288] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 2 election: Requested vote from peers 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:34.520771 346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:34.521529 424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:34.522974 556 raft_consensus.cc:504] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:34.523275 562 raft_consensus.cc:504] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:34.524537 424 raft_consensus.cc:2379] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 15218c77abfe47d9ab4765853cc3b2c4 in current term 2: Already voted for candidate 90b6685012e44e47a191e25ffc352954 in this term.
I20260808 01:08:34.524610 346 raft_consensus.cc:2379] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 15218c77abfe47d9ab4765853cc3b2c4 in current term 2: Already voted for candidate 97ca130eec6c4aa98ca7208cd35ed98a in this term.
I20260808 01:08:34.525487 562 leader_election.cc:288] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 2 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:34.525527 556 leader_election.cc:288] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 2 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:34.526984 346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:34.527134 32731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:34.527935 346 raft_consensus.cc:2379] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 90b6685012e44e47a191e25ffc352954 in current term 2: Already voted for candidate 97ca130eec6c4aa98ca7208cd35ed98a in this term.
I20260808 01:08:34.527963 32731 raft_consensus.cc:2379] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 90b6685012e44e47a191e25ffc352954 in current term 2: Already voted for candidate 15218c77abfe47d9ab4765853cc3b2c4 in this term.
I20260808 01:08:34.527972 32730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:34.529031 32730 raft_consensus.cc:2379] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 97ca130eec6c4aa98ca7208cd35ed98a in current term 2: Already voted for candidate 15218c77abfe47d9ab4765853cc3b2c4 in this term.
I20260808 01:08:34.528865 424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:34.529729 424 raft_consensus.cc:2379] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 97ca130eec6c4aa98ca7208cd35ed98a in current term 2: Already voted for candidate 90b6685012e44e47a191e25ffc352954 in this term.
I20260808 01:08:34.531060 32696 leader_election.cc:302] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4; no voters: 90b6685012e44e47a191e25ffc352954, 97ca130eec6c4aa98ca7208cd35ed98a
I20260808 01:08:34.531215 307 leader_election.cc:302] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97ca130eec6c4aa98ca7208cd35ed98a; no voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954
I20260808 01:08:34.531570 387 leader_election.cc:302] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90b6685012e44e47a191e25ffc352954; no voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a
I20260808 01:08:34.532099 562 raft_consensus.cc:2726] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:08:34.532437 561 raft_consensus.cc:2726] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:08:34.532675 556 raft_consensus.cc:2726] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260808 01:08:35.000854 32757 debug-util.cc:398] Leaking SignalData structure 0x7b08000ea060 after lost signal to thread 32596
W20260808 01:08:35.001801 32757 debug-util.cc:398] Leaking SignalData structure 0x7b08000ea740 after lost signal to thread 32663
W20260808 01:08:35.002523 32757 debug-util.cc:398] Leaking SignalData structure 0x7b080008d140 after lost signal to thread 32760
W20260808 01:08:35.003229 32757 debug-util.cc:398] Leaking SignalData structure 0x7b08000a63a0 after lost signal to thread 374
W20260808 01:08:35.003990 32757 debug-util.cc:398] Leaking SignalData structure 0x7b08001dc1c0 after lost signal to thread 450
W20260808 01:08:35.274294 32678 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260808 01:08:36.281488 32678 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:08:36.365505 563 raft_consensus.cc:482] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:08:36.366050 563 raft_consensus.cc:504] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:36.367817 563 leader_election.cc:288] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:36.368769 32730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4" is_pre_election: true
I20260808 01:08:36.369081 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" is_pre_election: true
I20260808 01:08:36.369419 32730 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 2.
I20260808 01:08:36.369827 348 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 2.
I20260808 01:08:36.370496 387 leader_election.cc:302] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954; no voters:
I20260808 01:08:36.371377 563 raft_consensus.cc:2781] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:08:36.371730 563 raft_consensus.cc:482] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:08:36.372131 563 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:08:36.377586 563 raft_consensus.cc:504] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:36.379294 563 leader_election.cc:288] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 3 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:36.380237 32730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:36.380446 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:36.380775 32730 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:08:36.381115 348 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:08:36.385962 32730 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 3.
I20260808 01:08:36.387032 387 leader_election.cc:302] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954; no voters:
I20260808 01:08:36.387781 563 raft_consensus.cc:2781] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:08:36.388586 348 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 3.
I20260808 01:08:36.389022 563 raft_consensus.cc:686] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 3 LEADER]: Becoming Leader. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Running, Role: LEADER
I20260808 01:08:36.390023 563 consensus_queue.cc:237] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:36.404068 32625 catalog_manager.cc:5495] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 reported cstate change: term changed from 0 to 3, leader changed from <none> to 90b6685012e44e47a191e25ffc352954 (127.31.212.67). New cstate: current_term: 3 leader_uuid: "90b6685012e44e47a191e25ffc352954" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: HEALTHY } } }
I20260808 01:08:36.500643 348 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea7e59a231e74d54b4859d7f52e8f0d1"
dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
mode: GRACEFUL
new_leader_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
from {username='slave'} at 127.0.0.1:33474
I20260808 01:08:36.501442 348 raft_consensus.cc:595] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Received request to transfer leadership to TS 15218c77abfe47d9ab4765853cc3b2c4
I20260808 01:08:36.526217 424 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143"
dest_uuid: "90b6685012e44e47a191e25ffc352954"
mode: GRACEFUL
new_leader_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
from {username='slave'} at 127.0.0.1:47556
I20260808 01:08:36.526835 424 raft_consensus.cc:595] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 1 LEADER]: Received request to transfer leadership to TS 15218c77abfe47d9ab4765853cc3b2c4
I20260808 01:08:36.532068 424 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26f5c989c2b7409c8ee0112f26936deb"
dest_uuid: "90b6685012e44e47a191e25ffc352954"
mode: GRACEFUL
new_leader_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
from {username='slave'} at 127.0.0.1:47556
I20260808 01:08:36.532667 424 raft_consensus.cc:595] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 1 LEADER]: Received request to transfer leadership to TS 15218c77abfe47d9ab4765853cc3b2c4
I20260808 01:08:36.538969 424 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bcbc69178831403785892df61938ee97"
dest_uuid: "90b6685012e44e47a191e25ffc352954"
mode: GRACEFUL
new_leader_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
from {username='slave'} at 127.0.0.1:47556
I20260808 01:08:36.539623 424 raft_consensus.cc:595] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 1 LEADER]: Received request to transfer leadership to TS 15218c77abfe47d9ab4765853cc3b2c4
I20260808 01:08:36.540609 565 raft_consensus.cc:979] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954: : Instructing follower 15218c77abfe47d9ab4765853cc3b2c4 to start an election
I20260808 01:08:36.541235 563 raft_consensus.cc:1067] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 1 LEADER]: Signalling peer 15218c77abfe47d9ab4765853cc3b2c4 to start an election
I20260808 01:08:36.544315 32730 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26f5c989c2b7409c8ee0112f26936deb"
dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
from {username='slave'} at 127.0.0.1:44778
I20260808 01:08:36.545099 32730 raft_consensus.cc:482] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:36.545534 32730 raft_consensus.cc:3037] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:36.550535 424 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58"
dest_uuid: "90b6685012e44e47a191e25ffc352954"
mode: GRACEFUL
new_leader_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
from {username='slave'} at 127.0.0.1:47556
I20260808 01:08:36.551218 424 raft_consensus.cc:595] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 3 LEADER]: Received request to transfer leadership to TS 97ca130eec6c4aa98ca7208cd35ed98a
I20260808 01:08:36.553200 32730 raft_consensus.cc:504] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:36.555162 32730 leader_election.cc:288] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 2 election: Requested vote from peers 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:36.556751 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f5c989c2b7409c8ee0112f26936deb" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:36.557353 348 raft_consensus.cc:3037] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:36.563354 348 raft_consensus.cc:2454] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15218c77abfe47d9ab4765853cc3b2c4 in term 2.
I20260808 01:08:36.564520 424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f5c989c2b7409c8ee0112f26936deb" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:36.565155 424 raft_consensus.cc:3032] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:08:36.565480 424 raft_consensus.cc:729] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Running, Role: LEADER
I20260808 01:08:36.566394 424 consensus_queue.cc:260] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:36.567430 424 raft_consensus.cc:3037] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:36.569092 346 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ca9ecb475d7143bd94ed0590fd4c5368"
dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
mode: GRACEFUL
new_leader_uuid: "90b6685012e44e47a191e25ffc352954"
from {username='slave'} at 127.0.0.1:33474
I20260808 01:08:36.569715 346 raft_consensus.cc:595] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Received request to transfer leadership to TS 90b6685012e44e47a191e25ffc352954
I20260808 01:08:36.569746 32696 leader_election.cc:302] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:36.575723 555 raft_consensus.cc:979] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a: : Instructing follower 15218c77abfe47d9ab4765853cc3b2c4 to start an election
I20260808 01:08:36.576295 555 raft_consensus.cc:1067] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Signalling peer 15218c77abfe47d9ab4765853cc3b2c4 to start an election
I20260808 01:08:36.576442 424 raft_consensus.cc:2454] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15218c77abfe47d9ab4765853cc3b2c4 in term 2.
I20260808 01:08:36.579064 32730 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea7e59a231e74d54b4859d7f52e8f0d1"
dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
from {username='slave'} at 127.0.0.1:44780
I20260808 01:08:36.579757 32730 raft_consensus.cc:482] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:36.580250 32730 raft_consensus.cc:3037] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:36.580420 576 raft_consensus.cc:2781] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:08:36.582082 576 raft_consensus.cc:686] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Becoming Leader. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Running, Role: LEADER
I20260808 01:08:36.583141 576 consensus_queue.cc:237] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:36.588167 32730 raft_consensus.cc:504] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:36.591087 32730 leader_election.cc:288] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 2 election: Requested vote from peers 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:36.591817 424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e59a231e74d54b4859d7f52e8f0d1" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:36.592497 424 raft_consensus.cc:3037] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:36.595041 32625 catalog_manager.cc:5495] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 reported cstate change: term changed from 1 to 2, leader changed from 90b6685012e44e47a191e25ffc352954 (127.31.212.67) to 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65). New cstate: current_term: 2 leader_uuid: "15218c77abfe47d9ab4765853cc3b2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } }
I20260808 01:08:36.596784 346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e59a231e74d54b4859d7f52e8f0d1" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:36.597455 346 raft_consensus.cc:3032] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:08:36.597826 346 raft_consensus.cc:729] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:36.599104 424 raft_consensus.cc:2454] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15218c77abfe47d9ab4765853cc3b2c4 in term 2.
I20260808 01:08:36.600440 32696 leader_election.cc:302] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954; no voters:
I20260808 01:08:36.601289 576 raft_consensus.cc:2781] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:08:36.601805 576 raft_consensus.cc:686] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Becoming Leader. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Running, Role: LEADER
I20260808 01:08:36.598707 346 consensus_queue.cc:260] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:36.602702 576 consensus_queue.cc:237] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:36.603787 346 raft_consensus.cc:3037] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:36.612488 346 raft_consensus.cc:2454] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15218c77abfe47d9ab4765853cc3b2c4 in term 2.
I20260808 01:08:36.616307 32625 catalog_manager.cc:5495] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 reported cstate change: term changed from 1 to 2, leader changed from 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66) to 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65). New cstate: current_term: 2 leader_uuid: "15218c77abfe47d9ab4765853cc3b2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: UNKNOWN } } }
I20260808 01:08:36.865973 565 raft_consensus.cc:979] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954: : Instructing follower 15218c77abfe47d9ab4765853cc3b2c4 to start an election
I20260808 01:08:36.866575 563 raft_consensus.cc:1067] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 1 LEADER]: Signalling peer 15218c77abfe47d9ab4765853cc3b2c4 to start an election
I20260808 01:08:36.868355 32731 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bcbc69178831403785892df61938ee97"
dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
from {username='slave'} at 127.0.0.1:44778
I20260808 01:08:36.868914 32731 raft_consensus.cc:482] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:36.869225 32731 raft_consensus.cc:3037] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:36.874724 32731 raft_consensus.cc:504] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:36.876699 32731 leader_election.cc:288] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 2 election: Requested vote from peers 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:36.878194 346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbc69178831403785892df61938ee97" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:36.878911 346 raft_consensus.cc:3037] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:36.878963 424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbc69178831403785892df61938ee97" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:36.879599 424 raft_consensus.cc:3032] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:08:36.880030 424 raft_consensus.cc:729] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Running, Role: LEADER
I20260808 01:08:36.880800 424 consensus_queue.cc:260] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:36.881901 424 raft_consensus.cc:3037] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:36.886853 424 raft_consensus.cc:2454] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15218c77abfe47d9ab4765853cc3b2c4 in term 2.
I20260808 01:08:36.887030 346 raft_consensus.cc:2454] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15218c77abfe47d9ab4765853cc3b2c4 in term 2.
I20260808 01:08:36.887966 32696 leader_election.cc:302] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954; no voters:
I20260808 01:08:36.888692 576 raft_consensus.cc:2781] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:08:36.889232 576 raft_consensus.cc:686] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Becoming Leader. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Running, Role: LEADER
I20260808 01:08:36.890167 576 consensus_queue.cc:237] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:36.898697 32625 catalog_manager.cc:5495] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 reported cstate change: term changed from 1 to 2, leader changed from 90b6685012e44e47a191e25ffc352954 (127.31.212.67) to 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65). New cstate: current_term: 2 leader_uuid: "15218c77abfe47d9ab4765853cc3b2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } }
I20260808 01:08:36.907132 346 raft_consensus.cc:1261] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 FOLLOWER]: Refusing update from remote peer 90b6685012e44e47a191e25ffc352954: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 1. (index mismatch)
I20260808 01:08:36.908525 563 consensus_queue.cc:1035] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:08:36.915740 565 raft_consensus.cc:979] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954: : Instructing follower 97ca130eec6c4aa98ca7208cd35ed98a to start an election
I20260808 01:08:36.916507 563 raft_consensus.cc:1067] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 3 LEADER]: Signalling peer 97ca130eec6c4aa98ca7208cd35ed98a to start an election
I20260808 01:08:36.918525 348 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58"
dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
from {username='slave'} at 127.0.0.1:43574
I20260808 01:08:36.919216 348 raft_consensus.cc:482] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:36.919629 348 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:08:36.924532 32731 raft_consensus.cc:1261] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 3 FOLLOWER]: Refusing update from remote peer 90b6685012e44e47a191e25ffc352954: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 1. (index mismatch)
I20260808 01:08:36.928974 563 consensus_queue.cc:1035] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:08:36.930475 563 raft_consensus.cc:979] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954: : Instructing follower 15218c77abfe47d9ab4765853cc3b2c4 to start an election
I20260808 01:08:36.931072 563 raft_consensus.cc:1067] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 1 LEADER]: Signalling peer 15218c77abfe47d9ab4765853cc3b2c4 to start an election
I20260808 01:08:36.930447 348 raft_consensus.cc:504] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:36.933462 346 raft_consensus.cc:1226] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 FOLLOWER]: Rejecting Update request from peer 90b6685012e44e47a191e25ffc352954 for earlier term 3. Current term is 4. Ops: []
I20260808 01:08:36.934398 32732 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143"
dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
from {username='slave'} at 127.0.0.1:44778
I20260808 01:08:36.934942 32732 raft_consensus.cc:482] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:36.935088 348 leader_election.cc:288] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 4 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:36.935297 32732 raft_consensus.cc:3037] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:36.936335 32733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: true dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:36.939306 424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: true dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:36.939977 424 raft_consensus.cc:3032] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:08:36.940297 424 raft_consensus.cc:729] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Running, Role: LEADER
I20260808 01:08:36.941102 424 consensus_queue.cc:260] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.1, Last appended by leader: 1, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:36.942061 424 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:08:36.943084 32732 raft_consensus.cc:504] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:36.946079 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:36.946759 348 raft_consensus.cc:3037] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:36.950278 424 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 4.
I20260808 01:08:36.951449 32732 leader_election.cc:288] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 2 election: Requested vote from peers 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:36.951524 307 leader_election.cc:302] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90b6685012e44e47a191e25ffc352954, 97ca130eec6c4aa98ca7208cd35ed98a; no voters: 15218c77abfe47d9ab4765853cc3b2c4
I20260808 01:08:36.952973 421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:36.954118 421 raft_consensus.cc:3032] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:08:36.954423 574 raft_consensus.cc:2781] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:08:36.954527 421 raft_consensus.cc:729] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Running, Role: LEADER
I20260808 01:08:36.954958 574 raft_consensus.cc:686] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 LEADER]: Becoming Leader. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:36.955317 421 consensus_queue.cc:260] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:36.956053 348 raft_consensus.cc:2454] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15218c77abfe47d9ab4765853cc3b2c4 in term 2.
I20260808 01:08:36.956700 421 raft_consensus.cc:3037] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:36.957224 32696 leader_election.cc:302] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:36.955792 574 consensus_queue.cc:237] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 3.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:36.962167 576 raft_consensus.cc:2781] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:08:36.962939 576 raft_consensus.cc:686] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Becoming Leader. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Running, Role: LEADER
I20260808 01:08:36.962984 421 raft_consensus.cc:2454] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15218c77abfe47d9ab4765853cc3b2c4 in term 2.
I20260808 01:08:36.963973 576 consensus_queue.cc:237] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:36.969274 32625 catalog_manager.cc:5495] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a reported cstate change: term changed from 3 to 4, leader changed from 90b6685012e44e47a191e25ffc352954 (127.31.212.67) to 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66). New cstate: current_term: 4 leader_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } }
I20260808 01:08:36.979686 32627 catalog_manager.cc:5495] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 reported cstate change: term changed from 1 to 2, leader changed from 90b6685012e44e47a191e25ffc352954 (127.31.212.67) to 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65). New cstate: current_term: 2 leader_uuid: "15218c77abfe47d9ab4765853cc3b2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } }
I20260808 01:08:37.028749 421 raft_consensus.cc:1261] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Refusing update from remote peer 15218c77abfe47d9ab4765853cc3b2c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:08:37.030123 576 consensus_queue.cc:1035] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:08:37.043536 348 raft_consensus.cc:1261] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Refusing update from remote peer 15218c77abfe47d9ab4765853cc3b2c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:08:37.045390 587 consensus_queue.cc:1035] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:08:37.125475 348 raft_consensus.cc:1261] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Refusing update from remote peer 15218c77abfe47d9ab4765853cc3b2c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:08:37.126956 587 consensus_queue.cc:1035] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:08:37.137441 555 raft_consensus.cc:979] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a: : Instructing follower 90b6685012e44e47a191e25ffc352954 to start an election
I20260808 01:08:37.137894 555 raft_consensus.cc:1067] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Signalling peer 90b6685012e44e47a191e25ffc352954 to start an election
I20260808 01:08:37.139465 421 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ca9ecb475d7143bd94ed0590fd4c5368"
dest_uuid: "90b6685012e44e47a191e25ffc352954"
from {username='slave'} at 127.0.0.1:43920
I20260808 01:08:37.140113 421 raft_consensus.cc:482] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:37.140365 424 raft_consensus.cc:1261] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Refusing update from remote peer 15218c77abfe47d9ab4765853cc3b2c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:08:37.140506 421 raft_consensus.cc:3037] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:37.142130 576 consensus_queue.cc:1035] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:08:37.148638 421 raft_consensus.cc:504] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:37.152720 32730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9ecb475d7143bd94ed0590fd4c5368" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:37.153443 32730 raft_consensus.cc:3037] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:37.157886 421 leader_election.cc:288] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 2 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:37.159762 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9ecb475d7143bd94ed0590fd4c5368" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:37.160588 348 raft_consensus.cc:3032] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:08:37.160988 348 raft_consensus.cc:729] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:37.161998 348 consensus_queue.cc:260] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:37.163007 32730 raft_consensus.cc:2454] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 2.
I20260808 01:08:37.163573 348 raft_consensus.cc:3037] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:37.164646 387 leader_election.cc:302] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954; no voters:
I20260808 01:08:37.168445 582 raft_consensus.cc:2781] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:08:37.169025 582 raft_consensus.cc:686] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [term 2 LEADER]: Becoming Leader. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Running, Role: LEADER
I20260808 01:08:37.169950 582 consensus_queue.cc:237] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:37.173698 348 raft_consensus.cc:2454] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 2.
I20260808 01:08:37.181253 32627 catalog_manager.cc:5495] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 reported cstate change: term changed from 1 to 2, leader changed from 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66) to 90b6685012e44e47a191e25ffc352954 (127.31.212.67). New cstate: current_term: 2 leader_uuid: "90b6685012e44e47a191e25ffc352954" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: HEALTHY } } }
I20260808 01:08:37.383566 424 raft_consensus.cc:1261] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Refusing update from remote peer 15218c77abfe47d9ab4765853cc3b2c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:08:37.385396 576 consensus_queue.cc:1035] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:08:37.398356 348 raft_consensus.cc:1261] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Refusing update from remote peer 15218c77abfe47d9ab4765853cc3b2c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:08:37.399946 587 consensus_queue.cc:1035] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:08:37.456477 348 raft_consensus.cc:1261] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Refusing update from remote peer 15218c77abfe47d9ab4765853cc3b2c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:08:37.461019 587 consensus_queue.cc:1035] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:08:37.475890 424 raft_consensus.cc:1261] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Refusing update from remote peer 15218c77abfe47d9ab4765853cc3b2c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:08:37.477582 587 consensus_queue.cc:1035] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:08:37.500541 32730 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:08:37.509105 32730 raft_consensus.cc:1261] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 4 FOLLOWER]: Refusing update from remote peer 97ca130eec6c4aa98ca7208cd35ed98a: Log matching property violated. Preceding OpId in replica: term: 3 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 01:08:37.510741 555 consensus_queue.cc:1035] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:08:37.525112 424 raft_consensus.cc:1261] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 4 FOLLOWER]: Refusing update from remote peer 97ca130eec6c4aa98ca7208cd35ed98a: Log matching property violated. Preceding OpId in replica: term: 3 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 01:08:37.526876 555 consensus_queue.cc:1035] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:08:37.670449 32730 raft_consensus.cc:1261] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Refusing update from remote peer 90b6685012e44e47a191e25ffc352954: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:08:37.671765 582 consensus_queue.cc:1035] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:08:37.684424 348 raft_consensus.cc:1261] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Refusing update from remote peer 90b6685012e44e47a191e25ffc352954: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:08:37.686170 602 consensus_queue.cc:1035] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:08:39.602491 32593 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260808 01:08:39.603543 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:39.624708 348 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58"
dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
mode: GRACEFUL
new_leader_uuid: "90b6685012e44e47a191e25ffc352954"
from {username='slave'} at 127.0.0.1:33482
I20260808 01:08:39.625519 348 raft_consensus.cc:595] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 LEADER]: Received request to transfer leadership to TS 90b6685012e44e47a191e25ffc352954
I20260808 01:08:39.640435 32730 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bcbc69178831403785892df61938ee97"
dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
mode: GRACEFUL
new_leader_uuid: "90b6685012e44e47a191e25ffc352954"
from {username='slave'} at 127.0.0.1:38320
I20260808 01:08:39.640998 32730 raft_consensus.cc:595] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Received request to transfer leadership to TS 90b6685012e44e47a191e25ffc352954
I20260808 01:08:39.642221 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 01:08:39.642783 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:08:39.702345 605 raft_consensus.cc:979] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a: : Instructing follower 90b6685012e44e47a191e25ffc352954 to start an election
I20260808 01:08:39.702965 605 raft_consensus.cc:1067] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 LEADER]: Signalling peer 90b6685012e44e47a191e25ffc352954 to start an election
I20260808 01:08:39.704922 424 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58"
dest_uuid: "90b6685012e44e47a191e25ffc352954"
from {username='slave'} at 127.0.0.1:43920
I20260808 01:08:39.705425 424 raft_consensus.cc:482] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:39.705731 424 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:08:39.711589 424 raft_consensus.cc:504] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:39.713544 424 leader_election.cc:288] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 5 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:39.714994 32730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:39.714988 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:39.715996 32730 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:08:39.716233 348 raft_consensus.cc:3032] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:08:39.716645 348 raft_consensus.cc:729] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:39.717466 348 consensus_queue.cc:260] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:39.718762 348 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:08:39.722139 32730 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 5.
I20260808 01:08:39.723793 348 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 5.
I20260808 01:08:39.723987 387 leader_election.cc:302] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954; no voters:
I20260808 01:08:39.725003 602 raft_consensus.cc:2781] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:08:39.726343 602 raft_consensus.cc:686] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 5 LEADER]: Becoming Leader. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Running, Role: LEADER
I20260808 01:08:39.727661 602 consensus_queue.cc:237] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:39.736485 32625 catalog_manager.cc:5495] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 reported cstate change: term changed from 4 to 5, leader changed from 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66) to 90b6685012e44e47a191e25ffc352954 (127.31.212.67). New cstate: current_term: 5 leader_uuid: "90b6685012e44e47a191e25ffc352954" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: HEALTHY } } }
I20260808 01:08:39.742339 577 raft_consensus.cc:979] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4: : Instructing follower 90b6685012e44e47a191e25ffc352954 to start an election
I20260808 01:08:39.743064 607 raft_consensus.cc:1067] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Signalling peer 90b6685012e44e47a191e25ffc352954 to start an election
I20260808 01:08:39.744923 424 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bcbc69178831403785892df61938ee97"
dest_uuid: "90b6685012e44e47a191e25ffc352954"
from {username='slave'} at 127.0.0.1:43904
I20260808 01:08:39.745604 424 raft_consensus.cc:482] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:39.746021 424 raft_consensus.cc:3037] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:08:39.753237 424 raft_consensus.cc:504] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:39.755048 424 leader_election.cc:288] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 3 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:39.755995 32730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbc69178831403785892df61938ee97" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:39.756400 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbc69178831403785892df61938ee97" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:39.756615 32730 raft_consensus.cc:3032] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:08:39.756948 32730 raft_consensus.cc:729] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Running, Role: LEADER
I20260808 01:08:39.756973 348 raft_consensus.cc:3037] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:08:39.757722 32730 consensus_queue.cc:260] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:39.758749 32730 raft_consensus.cc:3037] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:08:39.761977 348 raft_consensus.cc:2454] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 3.
I20260808 01:08:39.763108 387 leader_election.cc:302] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90b6685012e44e47a191e25ffc352954, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:39.763720 32730 raft_consensus.cc:2454] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 3.
I20260808 01:08:39.764029 602 raft_consensus.cc:2781] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:08:39.764679 602 raft_consensus.cc:686] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 3 LEADER]: Becoming Leader. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Running, Role: LEADER
I20260808 01:08:39.765497 602 consensus_queue.cc:237] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:39.773212 32625 catalog_manager.cc:5495] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 reported cstate change: term changed from 2 to 3, leader changed from 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65) to 90b6685012e44e47a191e25ffc352954 (127.31.212.67). New cstate: current_term: 3 leader_uuid: "90b6685012e44e47a191e25ffc352954" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: HEALTHY } } }
I20260808 01:08:40.236033 348 raft_consensus.cc:1261] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 5 FOLLOWER]: Refusing update from remote peer 90b6685012e44e47a191e25ffc352954: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 01:08:40.237317 602 consensus_queue.cc:1035] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:08:40.247328 32730 raft_consensus.cc:1261] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 5 FOLLOWER]: Refusing update from remote peer 90b6685012e44e47a191e25ffc352954: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 01:08:40.249164 602 consensus_queue.cc:1035] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:08:40.365571 348 raft_consensus.cc:1261] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 FOLLOWER]: Refusing update from remote peer 90b6685012e44e47a191e25ffc352954: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:08:40.366954 602 consensus_queue.cc:1035] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:08:40.377538 32730 raft_consensus.cc:1261] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 3 FOLLOWER]: Refusing update from remote peer 90b6685012e44e47a191e25ffc352954: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:08:40.379215 612 consensus_queue.cc:1035] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:08:40.643764 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:40.648526 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:08:40.648900 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:08:41.649827 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:41.654799 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:08:41.655193 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:08:42.656114 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:42.660972 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:08:42.661456 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:08:43.662403 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:43.668269 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:08:43.668720 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:08:44.669668 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:44.675071 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:08:44.675585 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:08:45.676708 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:45.681965 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:08:45.682422 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:08:46.683403 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:46.689339 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:08:46.690094 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:08:47.691097 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:47.696209 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:08:47.696659 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:08:48.698081 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:48.705147 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:08:48.705789 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:08:49.738595 32593 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260808 01:08:49.756470 32730 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea7e59a231e74d54b4859d7f52e8f0d1"
dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
mode: GRACEFUL
new_leader_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
from {username='slave'} at 127.0.0.1:40600
I20260808 01:08:49.757241 32730 raft_consensus.cc:595] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Received request to transfer leadership to TS 97ca130eec6c4aa98ca7208cd35ed98a
I20260808 01:08:49.762568 32730 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143"
dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
mode: GRACEFUL
new_leader_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
from {username='slave'} at 127.0.0.1:40600
I20260808 01:08:49.763077 32730 raft_consensus.cc:595] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Received request to transfer leadership to TS 97ca130eec6c4aa98ca7208cd35ed98a
I20260808 01:08:49.768237 32730 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26f5c989c2b7409c8ee0112f26936deb"
dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
mode: GRACEFUL
new_leader_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
from {username='slave'} at 127.0.0.1:40600
I20260808 01:08:49.768735 32730 raft_consensus.cc:595] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Received request to transfer leadership to TS 97ca130eec6c4aa98ca7208cd35ed98a
I20260808 01:08:49.774114 421 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bcbc69178831403785892df61938ee97"
dest_uuid: "90b6685012e44e47a191e25ffc352954"
mode: GRACEFUL
new_leader_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
from {username='slave'} at 127.0.0.1:47556
I20260808 01:08:49.774698 421 raft_consensus.cc:595] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 3 LEADER]: Received request to transfer leadership to TS 97ca130eec6c4aa98ca7208cd35ed98a
I20260808 01:08:49.783502 421 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58"
dest_uuid: "90b6685012e44e47a191e25ffc352954"
mode: GRACEFUL
new_leader_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
from {username='slave'} at 127.0.0.1:47556
I20260808 01:08:49.784185 421 raft_consensus.cc:595] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 5 LEADER]: Received request to transfer leadership to TS 97ca130eec6c4aa98ca7208cd35ed98a
I20260808 01:08:49.804726 607 raft_consensus.cc:979] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4: : Instructing follower 97ca130eec6c4aa98ca7208cd35ed98a to start an election
I20260808 01:08:49.805325 638 raft_consensus.cc:1067] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Signalling peer 97ca130eec6c4aa98ca7208cd35ed98a to start an election
I20260808 01:08:49.807004 348 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea7e59a231e74d54b4859d7f52e8f0d1"
dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
from {username='slave'} at 127.0.0.1:43558
I20260808 01:08:49.807512 348 raft_consensus.cc:482] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:49.807884 348 raft_consensus.cc:3037] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:08:49.813242 348 raft_consensus.cc:504] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:49.815490 348 leader_election.cc:288] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 3 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:49.816610 32730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e59a231e74d54b4859d7f52e8f0d1" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:49.816740 421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e59a231e74d54b4859d7f52e8f0d1" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:49.817250 32730 raft_consensus.cc:3032] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:08:49.817327 421 raft_consensus.cc:3037] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:08:49.817550 32730 raft_consensus.cc:729] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Running, Role: LEADER
I20260808 01:08:49.818325 32730 consensus_queue.cc:260] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:49.819643 32730 raft_consensus.cc:3037] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:08:49.823102 421 raft_consensus.cc:2454] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 3.
I20260808 01:08:49.824441 307 leader_election.cc:302] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90b6685012e44e47a191e25ffc352954, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:49.825452 32730 raft_consensus.cc:2454] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 3.
I20260808 01:08:49.825536 633 raft_consensus.cc:2781] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:08:49.826699 633 raft_consensus.cc:686] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 LEADER]: Becoming Leader. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:49.827579 633 consensus_queue.cc:237] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:49.834988 32625 catalog_manager.cc:5495] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a reported cstate change: term changed from 2 to 3, leader changed from 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65) to 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66). New cstate: current_term: 3 leader_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: HEALTHY } } }
I20260808 01:08:49.897241 602 raft_consensus.cc:979] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954: : Instructing follower 97ca130eec6c4aa98ca7208cd35ed98a to start an election
I20260808 01:08:49.897806 648 raft_consensus.cc:1067] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 3 LEADER]: Signalling peer 97ca130eec6c4aa98ca7208cd35ed98a to start an election
I20260808 01:08:49.899446 348 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bcbc69178831403785892df61938ee97"
dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
from {username='slave'} at 127.0.0.1:43574
I20260808 01:08:49.900007 348 raft_consensus.cc:482] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:49.900300 348 raft_consensus.cc:3037] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:08:49.905324 348 raft_consensus.cc:504] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:49.906731 348 leader_election.cc:288] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 4 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:49.907621 32730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbc69178831403785892df61938ee97" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:49.907691 421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbc69178831403785892df61938ee97" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:49.908243 32730 raft_consensus.cc:3037] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:08:49.908345 421 raft_consensus.cc:3032] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:08:49.908718 421 raft_consensus.cc:729] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Running, Role: LEADER
I20260808 01:08:49.909296 421 consensus_queue.cc:260] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:49.910298 421 raft_consensus.cc:3037] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:08:49.912730 32730 raft_consensus.cc:2454] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 4.
I20260808 01:08:49.913750 307 leader_election.cc:302] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:49.914389 633 raft_consensus.cc:2781] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:08:49.914775 633 raft_consensus.cc:686] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 LEADER]: Becoming Leader. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:49.915126 421 raft_consensus.cc:2454] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 4.
I20260808 01:08:49.915678 633 consensus_queue.cc:237] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:49.923162 32625 catalog_manager.cc:5495] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a reported cstate change: term changed from 3 to 4, leader changed from 90b6685012e44e47a191e25ffc352954 (127.31.212.67) to 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66). New cstate: current_term: 4 leader_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } }
I20260808 01:08:50.178038 607 raft_consensus.cc:979] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4: : Instructing follower 97ca130eec6c4aa98ca7208cd35ed98a to start an election
I20260808 01:08:50.178539 638 raft_consensus.cc:1067] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Signalling peer 97ca130eec6c4aa98ca7208cd35ed98a to start an election
I20260808 01:08:50.180212 348 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143"
dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
from {username='slave'} at 127.0.0.1:43558
I20260808 01:08:50.180783 348 raft_consensus.cc:482] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:50.181051 348 raft_consensus.cc:3037] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:08:50.186093 348 raft_consensus.cc:504] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:50.187750 348 leader_election.cc:288] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 3 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:50.188827 32730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:50.189142 421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:50.189601 32730 raft_consensus.cc:3032] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:08:50.189828 421 raft_consensus.cc:3037] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:08:50.190045 32730 raft_consensus.cc:729] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Running, Role: LEADER
I20260808 01:08:50.190963 32730 consensus_queue.cc:260] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:50.192364 32730 raft_consensus.cc:3037] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:08:50.197571 421 raft_consensus.cc:2454] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 3.
I20260808 01:08:50.198916 307 leader_election.cc:302] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90b6685012e44e47a191e25ffc352954, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:50.199426 32730 raft_consensus.cc:2454] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 3.
I20260808 01:08:50.199976 633 raft_consensus.cc:2781] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:08:50.200562 633 raft_consensus.cc:686] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 LEADER]: Becoming Leader. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:50.201442 633 consensus_queue.cc:237] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:50.208554 32625 catalog_manager.cc:5495] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a reported cstate change: term changed from 2 to 3, leader changed from 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65) to 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66). New cstate: current_term: 3 leader_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } }
I20260808 01:08:50.282527 421 raft_consensus.cc:1261] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [term 3 FOLLOWER]: Refusing update from remote peer 97ca130eec6c4aa98ca7208cd35ed98a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:08:50.283874 633 consensus_queue.cc:1035] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:08:50.294551 32730 raft_consensus.cc:1261] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 3 FOLLOWER]: Refusing update from remote peer 97ca130eec6c4aa98ca7208cd35ed98a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:08:50.297381 633 consensus_queue.cc:1035] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:08:50.300521 607 raft_consensus.cc:979] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4: : Instructing follower 97ca130eec6c4aa98ca7208cd35ed98a to start an election
I20260808 01:08:50.301179 607 raft_consensus.cc:1067] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Signalling peer 97ca130eec6c4aa98ca7208cd35ed98a to start an election
I20260808 01:08:50.303084 348 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26f5c989c2b7409c8ee0112f26936deb"
dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
from {username='slave'} at 127.0.0.1:43558
I20260808 01:08:50.304415 348 raft_consensus.cc:482] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:50.304929 348 raft_consensus.cc:3037] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:08:50.311239 348 raft_consensus.cc:504] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:50.312886 348 leader_election.cc:288] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 3 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:50.313714 32730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f5c989c2b7409c8ee0112f26936deb" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:50.314074 421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f5c989c2b7409c8ee0112f26936deb" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:50.314339 32730 raft_consensus.cc:3032] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:08:50.314760 421 raft_consensus.cc:3037] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:08:50.314725 32730 raft_consensus.cc:729] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Running, Role: LEADER
I20260808 01:08:50.315614 32730 consensus_queue.cc:260] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:50.316740 32730 raft_consensus.cc:3037] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:08:50.321483 421 raft_consensus.cc:2454] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 3.
I20260808 01:08:50.322542 307 leader_election.cc:302] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90b6685012e44e47a191e25ffc352954, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:50.322914 32730 raft_consensus.cc:2454] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 3.
I20260808 01:08:50.323663 656 raft_consensus.cc:2781] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:08:50.324156 656 raft_consensus.cc:686] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 LEADER]: Becoming Leader. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:50.325011 656 consensus_queue.cc:237] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:50.332777 32625 catalog_manager.cc:5495] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a reported cstate change: term changed from 2 to 3, leader changed from 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65) to 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66). New cstate: current_term: 3 leader_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } }
I20260808 01:08:50.333511 602 raft_consensus.cc:979] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954: : Instructing follower 97ca130eec6c4aa98ca7208cd35ed98a to start an election
I20260808 01:08:50.334002 648 raft_consensus.cc:1067] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 5 LEADER]: Signalling peer 97ca130eec6c4aa98ca7208cd35ed98a to start an election
I20260808 01:08:50.335733 348 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58"
dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
from {username='slave'} at 127.0.0.1:43574
I20260808 01:08:50.336432 348 raft_consensus.cc:482] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:50.336707 348 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:08:50.341740 348 raft_consensus.cc:504] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:50.343384 348 leader_election.cc:288] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 6 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:50.344173 32730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:50.344403 421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:50.344777 32730 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:08:50.344944 421 raft_consensus.cc:3032] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:08:50.345288 421 raft_consensus.cc:729] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Running, Role: LEADER
I20260808 01:08:50.346073 421 consensus_queue.cc:260] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:50.347038 421 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:08:50.351545 32730 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 6.
I20260808 01:08:50.353179 421 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ca130eec6c4aa98ca7208cd35ed98a in term 6.
I20260808 01:08:50.352905 307 leader_election.cc:302] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:50.355195 656 raft_consensus.cc:2781] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:08:50.355697 656 raft_consensus.cc:686] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 6 LEADER]: Becoming Leader. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:50.356587 656 consensus_queue.cc:237] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:50.364616 32625 catalog_manager.cc:5495] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a reported cstate change: term changed from 5 to 6, leader changed from 90b6685012e44e47a191e25ffc352954 (127.31.212.67) to 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66). New cstate: current_term: 6 leader_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } }
I20260808 01:08:50.522847 32730 raft_consensus.cc:1261] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 4 FOLLOWER]: Refusing update from remote peer 97ca130eec6c4aa98ca7208cd35ed98a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:08:50.524307 656 consensus_queue.cc:1035] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:08:50.528972 421 raft_consensus.cc:1261] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 4 FOLLOWER]: Refusing update from remote peer 97ca130eec6c4aa98ca7208cd35ed98a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:08:50.530402 656 consensus_queue.cc:1035] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:08:50.696347 32730 raft_consensus.cc:1261] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 3 FOLLOWER]: Refusing update from remote peer 97ca130eec6c4aa98ca7208cd35ed98a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:08:50.697659 656 consensus_queue.cc:1035] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:08:50.707427 421 raft_consensus.cc:1261] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 3 FOLLOWER]: Refusing update from remote peer 97ca130eec6c4aa98ca7208cd35ed98a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:08:50.709158 656 consensus_queue.cc:1035] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:08:50.931546 421 raft_consensus.cc:1261] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 3 FOLLOWER]: Refusing update from remote peer 97ca130eec6c4aa98ca7208cd35ed98a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:08:50.933007 641 consensus_queue.cc:1035] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:08:50.944603 32730 raft_consensus.cc:1261] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 3 FOLLOWER]: Refusing update from remote peer 97ca130eec6c4aa98ca7208cd35ed98a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:08:50.946344 667 consensus_queue.cc:1035] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:08:50.955574 421 raft_consensus.cc:1261] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 6 FOLLOWER]: Refusing update from remote peer 97ca130eec6c4aa98ca7208cd35ed98a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:08:50.956724 656 consensus_queue.cc:1035] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:08:50.966114 32730 raft_consensus.cc:1261] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 6 FOLLOWER]: Refusing update from remote peer 97ca130eec6c4aa98ca7208cd35ed98a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:08:50.967664 656 consensus_queue.cc:1035] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:08:52.826272 32593 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260808 01:08:52.827260 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:52.834579 348 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58"
dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
mode: GRACEFUL
new_leader_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
from {username='slave'} at 127.0.0.1:33482
I20260808 01:08:52.835189 348 raft_consensus.cc:595] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 6 LEADER]: Received request to transfer leadership to TS 15218c77abfe47d9ab4765853cc3b2c4
I20260808 01:08:52.838184 348 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7da895fe9b014f5d8f2acede20d67c53"
dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
mode: GRACEFUL
new_leader_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
from {username='slave'} at 127.0.0.1:33482
I20260808 01:08:52.838749 348 raft_consensus.cc:595] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Received request to transfer leadership to TS 15218c77abfe47d9ab4765853cc3b2c4
I20260808 01:08:52.841138 348 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0a4791d89904ee29ff78ef43c021b65"
dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
mode: GRACEFUL
new_leader_uuid: "90b6685012e44e47a191e25ffc352954"
from {username='slave'} at 127.0.0.1:33482
I20260808 01:08:52.841611 348 raft_consensus.cc:595] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Received request to transfer leadership to TS 90b6685012e44e47a191e25ffc352954
I20260808 01:08:52.843883 348 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143"
dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
mode: GRACEFUL
new_leader_uuid: "90b6685012e44e47a191e25ffc352954"
from {username='slave'} at 127.0.0.1:33482
I20260808 01:08:52.844470 348 raft_consensus.cc:595] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 LEADER]: Received request to transfer leadership to TS 90b6685012e44e47a191e25ffc352954
I20260808 01:08:52.846935 348 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea7e59a231e74d54b4859d7f52e8f0d1"
dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
mode: GRACEFUL
new_leader_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
from {username='slave'} at 127.0.0.1:33482
I20260808 01:08:52.847452 348 raft_consensus.cc:595] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 LEADER]: Received request to transfer leadership to TS 15218c77abfe47d9ab4765853cc3b2c4
I20260808 01:08:52.848520 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260808 01:08:52.848912 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:08:52.906181 633 raft_consensus.cc:979] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a: : Instructing follower 90b6685012e44e47a191e25ffc352954 to start an election
I20260808 01:08:52.906767 673 raft_consensus.cc:1067] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 LEADER]: Signalling peer 90b6685012e44e47a191e25ffc352954 to start an election
I20260808 01:08:52.908638 421 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143"
dest_uuid: "90b6685012e44e47a191e25ffc352954"
from {username='slave'} at 127.0.0.1:43920
I20260808 01:08:52.909160 421 raft_consensus.cc:482] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:52.909484 421 raft_consensus.cc:3037] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:08:52.910080 672 raft_consensus.cc:979] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a: : Instructing follower 15218c77abfe47d9ab4765853cc3b2c4 to start an election
I20260808 01:08:52.910553 673 raft_consensus.cc:1067] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Signalling peer 15218c77abfe47d9ab4765853cc3b2c4 to start an election
I20260808 01:08:52.912110 32730 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7da895fe9b014f5d8f2acede20d67c53"
dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
from {username='slave'} at 127.0.0.1:44780
I20260808 01:08:52.912616 32730 raft_consensus.cc:482] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:52.912957 32730 raft_consensus.cc:3037] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:52.916085 421 raft_consensus.cc:504] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:52.918573 421 leader_election.cc:288] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 4 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:52.919766 32732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:52.920521 32732 raft_consensus.cc:3037] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:08:52.919824 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d120a88b81634ee28ee1ec136a4d4143" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:52.922597 348 raft_consensus.cc:3032] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:08:52.922904 348 raft_consensus.cc:729] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:52.922979 32730 raft_consensus.cc:504] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:52.923642 348 consensus_queue.cc:260] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:52.924810 348 raft_consensus.cc:3037] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:08:52.925356 32730 leader_election.cc:288] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 2 election: Requested vote from peers 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:52.926643 32732 raft_consensus.cc:2454] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 4.
I20260808 01:08:52.926748 421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da895fe9b014f5d8f2acede20d67c53" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:52.927268 346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da895fe9b014f5d8f2acede20d67c53" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:52.927352 421 raft_consensus.cc:3037] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:52.928253 387 leader_election.cc:302] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954; no voters:
I20260808 01:08:52.928385 346 raft_consensus.cc:3032] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:08:52.928828 346 raft_consensus.cc:729] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:52.929211 671 raft_consensus.cc:2781] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:08:52.929623 346 consensus_queue.cc:260] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:52.930351 671 raft_consensus.cc:686] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 4 LEADER]: Becoming Leader. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Running, Role: LEADER
I20260808 01:08:52.931520 346 raft_consensus.cc:3037] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:52.931223 671 consensus_queue.cc:237] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:52.934492 348 raft_consensus.cc:2454] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 4.
I20260808 01:08:52.935889 421 raft_consensus.cc:2454] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15218c77abfe47d9ab4765853cc3b2c4 in term 2.
I20260808 01:08:52.937728 32696 leader_election.cc:302] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954; no voters:
I20260808 01:08:52.939060 346 raft_consensus.cc:2454] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15218c77abfe47d9ab4765853cc3b2c4 in term 2.
I20260808 01:08:52.940514 676 raft_consensus.cc:2781] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:08:52.941829 676 raft_consensus.cc:686] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Becoming Leader. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Running, Role: LEADER
I20260808 01:08:52.942567 32627 catalog_manager.cc:5495] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 reported cstate change: term changed from 3 to 4, leader changed from 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66) to 90b6685012e44e47a191e25ffc352954 (127.31.212.67). New cstate: current_term: 4 leader_uuid: "90b6685012e44e47a191e25ffc352954" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: HEALTHY } } }
I20260808 01:08:52.942718 676 consensus_queue.cc:237] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:52.950667 32625 catalog_manager.cc:5495] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 reported cstate change: term changed from 1 to 2, leader changed from 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66) to 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65). New cstate: current_term: 2 leader_uuid: "15218c77abfe47d9ab4765853cc3b2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } }
I20260808 01:08:53.115120 673 raft_consensus.cc:979] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a: : Instructing follower 90b6685012e44e47a191e25ffc352954 to start an election
I20260808 01:08:53.115669 672 raft_consensus.cc:1067] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Signalling peer 90b6685012e44e47a191e25ffc352954 to start an election
I20260808 01:08:53.117314 421 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b0a4791d89904ee29ff78ef43c021b65"
dest_uuid: "90b6685012e44e47a191e25ffc352954"
from {username='slave'} at 127.0.0.1:43920
I20260808 01:08:53.117905 421 raft_consensus.cc:482] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:53.118212 421 raft_consensus.cc:3037] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:53.122836 421 raft_consensus.cc:504] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:53.124574 421 leader_election.cc:288] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 2 election: Requested vote from peers 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:53.125551 32732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a4791d89904ee29ff78ef43c021b65" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
I20260808 01:08:53.125886 346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a4791d89904ee29ff78ef43c021b65" candidate_uuid: "90b6685012e44e47a191e25ffc352954" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:53.126143 32732 raft_consensus.cc:3037] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:53.126500 346 raft_consensus.cc:3032] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:08:53.126780 346 raft_consensus.cc:729] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:53.127398 346 consensus_queue.cc:260] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:53.128427 346 raft_consensus.cc:3037] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:08:53.131423 32732 raft_consensus.cc:2454] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 2.
I20260808 01:08:53.132671 387 leader_election.cc:302] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954; no voters:
I20260808 01:08:53.133617 671 raft_consensus.cc:2781] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:08:53.133879 346 raft_consensus.cc:2454] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b6685012e44e47a191e25ffc352954 in term 2.
I20260808 01:08:53.134140 671 raft_consensus.cc:686] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [term 2 LEADER]: Becoming Leader. State: Replica: 90b6685012e44e47a191e25ffc352954, State: Running, Role: LEADER
I20260808 01:08:53.135144 671 consensus_queue.cc:237] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:53.143086 32625 catalog_manager.cc:5495] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 reported cstate change: term changed from 1 to 2, leader changed from 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66) to 90b6685012e44e47a191e25ffc352954 (127.31.212.67). New cstate: current_term: 2 leader_uuid: "90b6685012e44e47a191e25ffc352954" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: HEALTHY } } }
I20260808 01:08:53.163674 673 raft_consensus.cc:979] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a: : Instructing follower 15218c77abfe47d9ab4765853cc3b2c4 to start an election
I20260808 01:08:53.164333 673 raft_consensus.cc:1067] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 6 LEADER]: Signalling peer 15218c77abfe47d9ab4765853cc3b2c4 to start an election
I20260808 01:08:53.165882 32732 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58"
dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
from {username='slave'} at 127.0.0.1:44780
I20260808 01:08:53.166442 32732 raft_consensus.cc:482] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:53.166787 32732 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:08:53.171499 32732 raft_consensus.cc:504] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:53.173957 32732 leader_election.cc:288] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 7 election: Requested vote from peers 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833), 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685)
I20260808 01:08:53.174116 346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:53.174767 346 raft_consensus.cc:3032] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:08:53.175081 346 raft_consensus.cc:729] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:53.175927 346 consensus_queue.cc:260] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:53.176151 421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cda7aa230904dc6bff9da8fba188d58" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:53.176749 421 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:08:53.177127 346 raft_consensus.cc:3037] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:08:53.183421 346 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15218c77abfe47d9ab4765853cc3b2c4 in term 7.
I20260808 01:08:53.183421 421 raft_consensus.cc:2454] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15218c77abfe47d9ab4765853cc3b2c4 in term 7.
I20260808 01:08:53.184777 32696 leader_election.cc:302] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 97ca130eec6c4aa98ca7208cd35ed98a; no voters:
I20260808 01:08:53.185449 676 raft_consensus.cc:2781] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:08:53.185909 676 raft_consensus.cc:686] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 7 LEADER]: Becoming Leader. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Running, Role: LEADER
I20260808 01:08:53.186712 676 consensus_queue.cc:237] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } }
I20260808 01:08:53.194290 32625 catalog_manager.cc:5495] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 reported cstate change: term changed from 6 to 7, leader changed from 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66) to 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65). New cstate: current_term: 7 leader_uuid: "15218c77abfe47d9ab4765853cc3b2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } }
I20260808 01:08:53.303719 672 raft_consensus.cc:979] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a: : Instructing follower 15218c77abfe47d9ab4765853cc3b2c4 to start an election
I20260808 01:08:53.304263 672 raft_consensus.cc:1067] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 LEADER]: Signalling peer 15218c77abfe47d9ab4765853cc3b2c4 to start an election
I20260808 01:08:53.305721 32732 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea7e59a231e74d54b4859d7f52e8f0d1"
dest_uuid: "15218c77abfe47d9ab4765853cc3b2c4"
from {username='slave'} at 127.0.0.1:44780
I20260808 01:08:53.306245 32732 raft_consensus.cc:482] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:08:53.306552 32732 raft_consensus.cc:3037] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:08:53.311596 32732 raft_consensus.cc:504] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:53.313308 32732 leader_election.cc:288] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 4 election: Requested vote from peers 90b6685012e44e47a191e25ffc352954 (127.31.212.67:42685), 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66:34833)
I20260808 01:08:53.314280 421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e59a231e74d54b4859d7f52e8f0d1" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "90b6685012e44e47a191e25ffc352954"
I20260808 01:08:53.314757 421 raft_consensus.cc:3037] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:08:53.314560 346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7e59a231e74d54b4859d7f52e8f0d1" candidate_uuid: "15218c77abfe47d9ab4765853cc3b2c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "97ca130eec6c4aa98ca7208cd35ed98a"
I20260808 01:08:53.315317 346 raft_consensus.cc:3032] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:08:53.315559 346 raft_consensus.cc:729] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 97ca130eec6c4aa98ca7208cd35ed98a, State: Running, Role: LEADER
I20260808 01:08:53.316246 346 consensus_queue.cc:260] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:53.317216 346 raft_consensus.cc:3037] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:08:53.319404 421 raft_consensus.cc:2454] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15218c77abfe47d9ab4765853cc3b2c4 in term 4.
I20260808 01:08:53.320358 32696 leader_election.cc:302] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15218c77abfe47d9ab4765853cc3b2c4, 90b6685012e44e47a191e25ffc352954; no voters:
I20260808 01:08:53.321199 676 raft_consensus.cc:2781] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:08:53.321678 676 raft_consensus.cc:686] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 4 LEADER]: Becoming Leader. State: Replica: 15218c77abfe47d9ab4765853cc3b2c4, State: Running, Role: LEADER
I20260808 01:08:53.322916 346 raft_consensus.cc:2454] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15218c77abfe47d9ab4765853cc3b2c4 in term 4.
I20260808 01:08:53.322507 676 consensus_queue.cc:237] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } }
I20260808 01:08:53.330046 32625 catalog_manager.cc:5495] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 reported cstate change: term changed from 3 to 4, leader changed from 97ca130eec6c4aa98ca7208cd35ed98a (127.31.212.66) to 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65). New cstate: current_term: 4 leader_uuid: "15218c77abfe47d9ab4765853cc3b2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 } health_report { overall_health: UNKNOWN } } }
I20260808 01:08:53.334697 32732 raft_consensus.cc:1261] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 4 FOLLOWER]: Refusing update from remote peer 90b6685012e44e47a191e25ffc352954: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:08:53.336100 671 consensus_queue.cc:1035] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:08:53.345377 346 raft_consensus.cc:1261] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 FOLLOWER]: Refusing update from remote peer 90b6685012e44e47a191e25ffc352954: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:08:53.347261 671 consensus_queue.cc:1035] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:08:53.488252 346 raft_consensus.cc:1261] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Refusing update from remote peer 15218c77abfe47d9ab4765853cc3b2c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:08:53.489461 676 consensus_queue.cc:1035] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:08:53.498584 421 raft_consensus.cc:1261] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Refusing update from remote peer 15218c77abfe47d9ab4765853cc3b2c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:08:53.500679 692 consensus_queue.cc:1035] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:08:53.548008 32732 raft_consensus.cc:1261] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Refusing update from remote peer 90b6685012e44e47a191e25ffc352954: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:08:53.549430 671 consensus_queue.cc:1035] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15218c77abfe47d9ab4765853cc3b2c4" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:08:53.559945 346 raft_consensus.cc:1261] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Refusing update from remote peer 90b6685012e44e47a191e25ffc352954: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:08:53.561779 671 consensus_queue.cc:1035] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:08:53.742467 421 raft_consensus.cc:1261] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 7 FOLLOWER]: Refusing update from remote peer 15218c77abfe47d9ab4765853cc3b2c4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 01:08:53.743649 692 consensus_queue.cc:1035] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:08:53.755077 346 raft_consensus.cc:1261] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 7 FOLLOWER]: Refusing update from remote peer 15218c77abfe47d9ab4765853cc3b2c4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 01:08:53.756943 698 consensus_queue.cc:1035] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:08:53.758332 346 raft_consensus.cc:1261] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 FOLLOWER]: Refusing update from remote peer 15218c77abfe47d9ab4765853cc3b2c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:08:53.760051 698 consensus_queue.cc:1035] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ca130eec6c4aa98ca7208cd35ed98a" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:08:53.770042 421 raft_consensus.cc:1261] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [term 4 FOLLOWER]: Refusing update from remote peer 15218c77abfe47d9ab4765853cc3b2c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:08:53.771725 676 consensus_queue.cc:1035] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b6685012e44e47a191e25ffc352954" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:08:53.849763 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:53.855547 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:08:53.856110 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:08:54.856916 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:54.861914 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:08:54.862396 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:08:55.863309 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:55.868240 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:08:55.868741 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:08:56.869659 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:56.874962 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:08:56.875471 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:08:57.963743 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:57.969805 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:08:57.970299 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260808 01:08:58.181581 447 debug-util.cc:398] Leaking SignalData structure 0x7b0800173060 after lost signal to thread 32596
W20260808 01:08:58.182438 447 debug-util.cc:398] Leaking SignalData structure 0x7b0800173740 after lost signal to thread 32663
W20260808 01:08:58.183295 447 debug-util.cc:398] Leaking SignalData structure 0x7b08000afa60 after lost signal to thread 32760
W20260808 01:08:58.184093 447 debug-util.cc:398] Leaking SignalData structure 0x7b08001ba980 after lost signal to thread 374
W20260808 01:08:58.184782 447 debug-util.cc:398] Leaking SignalData structure 0x7b08001c86e0 after lost signal to thread 450
I20260808 01:08:58.971659 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:58.980783 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:08:58.981280 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:08:59.982185 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:08:59.987732 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:08:59.988337 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:00.989363 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:00.994942 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:00.995416 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:01.996384 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:02.001338 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:02.001927 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:03.032882 32593 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260808 01:09:03.033557 32593 tablet_server.cc:178] TabletServer@127.31.212.65:0 shutting down...
I20260808 01:09:03.068305 32593 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260808 01:09:03.069084 32593 tablet_replica.cc:330] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4: stopping tablet replica
I20260808 01:09:03.069883 32593 raft_consensus.cc:2229] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.070456 32593 raft_consensus.cc:2258] T ca9ecb475d7143bd94ed0590fd4c5368 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.073101 32593 tablet_replica.cc:330] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 15218c77abfe47d9ab4765853cc3b2c4: stopping tablet replica
I20260808 01:09:03.073649 32593 raft_consensus.cc:2229] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.074123 32593 raft_consensus.cc:2258] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 15218c77abfe47d9ab4765853cc3b2c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.075809 32593 tablet_replica.cc:330] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4: stopping tablet replica
I20260808 01:09:03.076375 32593 raft_consensus.cc:2229] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 7 LEADER]: Raft consensus shutting down.
I20260808 01:09:03.077217 32593 raft_consensus.cc:2258] T 6cda7aa230904dc6bff9da8fba188d58 P 15218c77abfe47d9ab4765853cc3b2c4 [term 7 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.078948 32593 tablet_replica.cc:330] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4: stopping tablet replica
I20260808 01:09:03.079435 32593 raft_consensus.cc:2229] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.079912 32593 raft_consensus.cc:2258] T b0a4791d89904ee29ff78ef43c021b65 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.081662 32593 tablet_replica.cc:330] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4: stopping tablet replica
I20260808 01:09:03.082310 32593 raft_consensus.cc:2229] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.082929 32593 raft_consensus.cc:2258] T 26f5c989c2b7409c8ee0112f26936deb P 15218c77abfe47d9ab4765853cc3b2c4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.084873 32593 tablet_replica.cc:330] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4: stopping tablet replica
I20260808 01:09:03.085407 32593 raft_consensus.cc:2229] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 LEADER]: Raft consensus shutting down.
I20260808 01:09:03.086227 32593 raft_consensus.cc:2258] T 7da895fe9b014f5d8f2acede20d67c53 P 15218c77abfe47d9ab4765853cc3b2c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.088102 32593 tablet_replica.cc:330] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4: stopping tablet replica
I20260808 01:09:03.088699 32593 raft_consensus.cc:2229] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.089207 32593 raft_consensus.cc:2258] T bcbc69178831403785892df61938ee97 P 15218c77abfe47d9ab4765853cc3b2c4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.091518 32593 tablet_replica.cc:330] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4: stopping tablet replica
I20260808 01:09:03.092096 32593 raft_consensus.cc:2229] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 4 LEADER]: Raft consensus shutting down.
I20260808 01:09:03.092945 32593 raft_consensus.cc:2258] T ea7e59a231e74d54b4859d7f52e8f0d1 P 15218c77abfe47d9ab4765853cc3b2c4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.094691 32593 tablet_replica.cc:330] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4: stopping tablet replica
I20260808 01:09:03.095165 32593 raft_consensus.cc:2229] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.095676 32593 raft_consensus.cc:2258] T d120a88b81634ee28ee1ec136a4d4143 P 15218c77abfe47d9ab4765853cc3b2c4 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260808 01:09:03.107892 307 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:09:03.112591 307 consensus_peers.cc:487] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a -> Peer 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753): Couldn't send request to peer 15218c77abfe47d9ab4765853cc3b2c4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:09:03.200845 387 consensus_peers.cc:487] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 -> Peer 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753): Couldn't send request to peer 15218c77abfe47d9ab4765853cc3b2c4. Status: Network error: Client connection negotiation failed: client connection to 127.31.212.65:40753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:03.217106 32593 tablet_server.cc:194] TabletServer@127.31.212.65:0 shutdown complete.
I20260808 01:09:03.237313 32593 tablet_server.cc:178] TabletServer@127.31.212.66:0 shutting down...
I20260808 01:09:03.271551 32593 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260808 01:09:03.272305 32593 tablet_replica.cc:330] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a: stopping tablet replica
I20260808 01:09:03.273288 32593 raft_consensus.cc:2229] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 LEADER]: Raft consensus shutting down.
I20260808 01:09:03.274899 32593 raft_consensus.cc:2258] T 26f5c989c2b7409c8ee0112f26936deb P 97ca130eec6c4aa98ca7208cd35ed98a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.276890 32593 tablet_replica.cc:330] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a: stopping tablet replica
I20260808 01:09:03.277451 32593 raft_consensus.cc:2229] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.277835 32593 raft_consensus.cc:2258] T d120a88b81634ee28ee1ec136a4d4143 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.279458 32593 tablet_replica.cc:330] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a: stopping tablet replica
I20260808 01:09:03.280012 32593 raft_consensus.cc:2229] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.280402 32593 raft_consensus.cc:2258] T ca9ecb475d7143bd94ed0590fd4c5368 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.281972 32593 tablet_replica.cc:330] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a: stopping tablet replica
I20260808 01:09:03.282473 32593 raft_consensus.cc:2229] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.282925 32593 raft_consensus.cc:2258] T b0a4791d89904ee29ff78ef43c021b65 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.284547 32593 tablet_replica.cc:330] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a: stopping tablet replica
I20260808 01:09:03.285043 32593 raft_consensus.cc:2229] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 LEADER]: Raft consensus shutting down.
I20260808 01:09:03.285827 32593 raft_consensus.cc:2258] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 97ca130eec6c4aa98ca7208cd35ed98a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.287488 32593 tablet_replica.cc:330] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a: stopping tablet replica
I20260808 01:09:03.288082 32593 raft_consensus.cc:2229] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 7 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.288641 32593 raft_consensus.cc:2258] T 6cda7aa230904dc6bff9da8fba188d58 P 97ca130eec6c4aa98ca7208cd35ed98a [term 7 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.290467 32593 tablet_replica.cc:330] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a: stopping tablet replica
I20260808 01:09:03.290958 32593 raft_consensus.cc:2229] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 LEADER]: Raft consensus shutting down.
I20260808 01:09:03.291749 32593 raft_consensus.cc:2258] T bcbc69178831403785892df61938ee97 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.293625 32593 tablet_replica.cc:330] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a: stopping tablet replica
I20260808 01:09:03.294099 32593 raft_consensus.cc:2229] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.294514 32593 raft_consensus.cc:2258] T 7da895fe9b014f5d8f2acede20d67c53 P 97ca130eec6c4aa98ca7208cd35ed98a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.296625 32593 tablet_replica.cc:330] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a: stopping tablet replica
I20260808 01:09:03.297091 32593 raft_consensus.cc:2229] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.297591 32593 raft_consensus.cc:2258] T ea7e59a231e74d54b4859d7f52e8f0d1 P 97ca130eec6c4aa98ca7208cd35ed98a [term 4 FOLLOWER]: Raft consensus is shut down!
W20260808 01:09:03.299151 387 consensus_peers.cc:487] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 -> Peer 15218c77abfe47d9ab4765853cc3b2c4 (127.31.212.65:40753): Couldn't send request to peer 15218c77abfe47d9ab4765853cc3b2c4. Status: Network error: Client connection negotiation failed: client connection to 127.31.212.65:40753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:03.326959 32593 tablet_server.cc:194] TabletServer@127.31.212.66:0 shutdown complete.
I20260808 01:09:03.345692 32593 tablet_server.cc:178] TabletServer@127.31.212.67:0 shutting down...
I20260808 01:09:03.376899 32593 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260808 01:09:03.377657 32593 tablet_replica.cc:330] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954: stopping tablet replica
I20260808 01:09:03.378369 32593 raft_consensus.cc:2229] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 4 LEADER]: Raft consensus shutting down.
I20260808 01:09:03.379213 32593 raft_consensus.cc:2258] T d120a88b81634ee28ee1ec136a4d4143 P 90b6685012e44e47a191e25ffc352954 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.381099 32593 tablet_replica.cc:330] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954: stopping tablet replica
I20260808 01:09:03.381639 32593 raft_consensus.cc:2229] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.382067 32593 raft_consensus.cc:2258] T 7da895fe9b014f5d8f2acede20d67c53 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.383903 32593 tablet_replica.cc:330] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954: stopping tablet replica
I20260808 01:09:03.384423 32593 raft_consensus.cc:2229] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.384842 32593 raft_consensus.cc:2258] T bcbc69178831403785892df61938ee97 P 90b6685012e44e47a191e25ffc352954 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.386699 32593 tablet_replica.cc:330] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954: stopping tablet replica
I20260808 01:09:03.387256 32593 raft_consensus.cc:2229] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [term 2 LEADER]: Raft consensus shutting down.
I20260808 01:09:03.388079 32593 raft_consensus.cc:2258] T b0a4791d89904ee29ff78ef43c021b65 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.389851 32593 tablet_replica.cc:330] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954: stopping tablet replica
I20260808 01:09:03.390381 32593 raft_consensus.cc:2229] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.390815 32593 raft_consensus.cc:2258] T ea7e59a231e74d54b4859d7f52e8f0d1 P 90b6685012e44e47a191e25ffc352954 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.392530 32593 tablet_replica.cc:330] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954: stopping tablet replica
I20260808 01:09:03.393030 32593 raft_consensus.cc:2229] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.393465 32593 raft_consensus.cc:2258] T 9f8a848bc0b4438b8ae3e5a5ff59f3b6 P 90b6685012e44e47a191e25ffc352954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.395159 32593 tablet_replica.cc:330] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954: stopping tablet replica
I20260808 01:09:03.395625 32593 raft_consensus.cc:2229] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [term 2 LEADER]: Raft consensus shutting down.
I20260808 01:09:03.396440 32593 raft_consensus.cc:2258] T ca9ecb475d7143bd94ed0590fd4c5368 P 90b6685012e44e47a191e25ffc352954 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.398061 32593 tablet_replica.cc:330] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954: stopping tablet replica
I20260808 01:09:03.398535 32593 raft_consensus.cc:2229] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 7 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.398954 32593 raft_consensus.cc:2258] T 6cda7aa230904dc6bff9da8fba188d58 P 90b6685012e44e47a191e25ffc352954 [term 7 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.400614 32593 tablet_replica.cc:330] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954: stopping tablet replica
I20260808 01:09:03.401062 32593 raft_consensus.cc:2229] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:03.401500 32593 raft_consensus.cc:2258] T 26f5c989c2b7409c8ee0112f26936deb P 90b6685012e44e47a191e25ffc352954 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.444890 32593 tablet_server.cc:194] TabletServer@127.31.212.67:0 shutdown complete.
I20260808 01:09:03.463503 32593 master.cc:537] Master@127.31.212.126:39845 shutting down...
I20260808 01:09:03.532636 32685 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Closing
I20260808 01:09:03.826524 32593 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [term 1 LEADER]: Raft consensus shutting down.
I20260808 01:09:03.827392 32593 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:03.827909 32593 tablet_replica.cc:330] T 00000000000000000000000000000000 P 643de1b5abe34a92afee3d3ced1b7c6d: stopping tablet replica
I20260808 01:09:03.847054 32593 master.cc:552] Master@127.31.212.126:39845 shutdown complete.
15218c77abfe47d9ab4765853cc3b2c4 4
90b6685012e44e47a191e25ffc352954 1
97ca130eec6c4aa98ca7208cd35ed98a 4
15218c77abfe47d9ab4765853cc3b2c4 3
90b6685012e44e47a191e25ffc352954 3
97ca130eec6c4aa98ca7208cd35ed98a 3
15218c77abfe47d9ab4765853cc3b2c4 0
90b6685012e44e47a191e25ffc352954 1
97ca130eec6c4aa98ca7208cd35ed98a 8
15218c77abfe47d9ab4765853cc3b2c4 3
90b6685012e44e47a191e25ffc352954 3
97ca130eec6c4aa98ca7208cd35ed98a 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (35464 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260808 01:09:03.894609 32593 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.31.212.126:36597
I20260808 01:09:03.895905 32593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 01:09:03.902729 718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:09:03.902837 719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:09:03.903333 721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:09:03.904191 32593 server_base.cc:1034] running on GCE node
I20260808 01:09:03.905400 32593 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:09:03.905606 32593 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:09:03.905730 32593 hybrid_clock.cc:648] HybridClock initialized: now 1786151343905721 us; error 0 us; skew 500 ppm
I20260808 01:09:03.906239 32593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:09:03.909559 32593 webserver.cc:460] Webserver started at http://127.31.212.126:38077/ using document root <none> and password file <none>
I20260808 01:09:03.910171 32593 fs_manager.cc:359] Metadata directory not provided
I20260808 01:09:03.910353 32593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:09:03.910763 32593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:09:03.912582 32593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/master-0-root/instance:
uuid: "375c7e95bb3b47249db008b5db57a83a"
format_stamp: "Formatted at 2026-08-08 01:09:03 on dist-test-slave-mvvj"
I20260808 01:09:03.918233 32593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260808 01:09:03.922227 726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:03.923134 32593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 01:09:03.923508 32593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/master-0-root
uuid: "375c7e95bb3b47249db008b5db57a83a"
format_stamp: "Formatted at 2026-08-08 01:09:03 on dist-test-slave-mvvj"
I20260808 01:09:03.923790 32593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:09:03.949847 32593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 01:09:03.951164 32593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:09:03.992565 32593 rpc_server.cc:307] RPC server started. Bound to: 127.31.212.126:36597
I20260808 01:09:03.992693 779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.212.126:36597 every 8 connection(s)
I20260808 01:09:03.996871 780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.006902 780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a: Bootstrap starting.
I20260808 01:09:04.011369 780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:04.015885 780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a: No bootstrap required, opened a new log
I20260808 01:09:04.017935 780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375c7e95bb3b47249db008b5db57a83a" member_type: VOTER }
I20260808 01:09:04.018426 780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:04.018599 780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375c7e95bb3b47249db008b5db57a83a, State: Initialized, Role: FOLLOWER
I20260808 01:09:04.019196 780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375c7e95bb3b47249db008b5db57a83a" member_type: VOTER }
I20260808 01:09:04.019642 780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 01:09:04.019870 780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 01:09:04.020195 780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:04.024322 780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375c7e95bb3b47249db008b5db57a83a" member_type: VOTER }
I20260808 01:09:04.024992 780 leader_election.cc:302] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 375c7e95bb3b47249db008b5db57a83a; no voters:
I20260808 01:09:04.026343 780 leader_election.cc:288] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 01:09:04.026763 783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:04.028185 783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [term 1 LEADER]: Becoming Leader. State: Replica: 375c7e95bb3b47249db008b5db57a83a, State: Running, Role: LEADER
I20260808 01:09:04.028914 783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375c7e95bb3b47249db008b5db57a83a" member_type: VOTER }
I20260808 01:09:04.029402 780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 01:09:04.034585 784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "375c7e95bb3b47249db008b5db57a83a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375c7e95bb3b47249db008b5db57a83a" member_type: VOTER } }
I20260808 01:09:04.035573 784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [sys.catalog]: This master's current role is: LEADER
I20260808 01:09:04.035247 785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 375c7e95bb3b47249db008b5db57a83a. Latest consensus state: current_term: 1 leader_uuid: "375c7e95bb3b47249db008b5db57a83a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375c7e95bb3b47249db008b5db57a83a" member_type: VOTER } }
I20260808 01:09:04.036845 785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [sys.catalog]: This master's current role is: LEADER
I20260808 01:09:04.046600 32593 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260808 01:09:04.046633 792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 01:09:04.051578 792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 01:09:04.060701 792 catalog_manager.cc:1347] Generated new cluster ID: 50fe737222d24fa88c5d0114041c8dfe
I20260808 01:09:04.060990 792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 01:09:04.080286 792 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 01:09:04.081967 792 catalog_manager.cc:1504] Loading token signing keys...
I20260808 01:09:04.096340 792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a: Generated new TSK 0
I20260808 01:09:04.097033 792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 01:09:04.113586 32593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 01:09:04.120721 803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:09:04.122069 804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:09:04.124297 32593 server_base.cc:1034] running on GCE node
W20260808 01:09:04.124517 806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:09:04.125954 32593 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:09:04.126210 32593 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:09:04.126394 32593 hybrid_clock.cc:648] HybridClock initialized: now 1786151344126373 us; error 0 us; skew 500 ppm
I20260808 01:09:04.126989 32593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:09:04.129801 32593 webserver.cc:460] Webserver started at http://127.31.212.65:45141/ using document root <none> and password file <none>
I20260808 01:09:04.130393 32593 fs_manager.cc:359] Metadata directory not provided
I20260808 01:09:04.130640 32593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:09:04.130937 32593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:09:04.132383 32593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/ts-0-root/instance:
uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
format_stamp: "Formatted at 2026-08-08 01:09:04 on dist-test-slave-mvvj"
I20260808 01:09:04.137548 32593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 01:09:04.141765 811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:04.142726 32593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 01:09:04.143013 32593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/ts-0-root
uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
format_stamp: "Formatted at 2026-08-08 01:09:04 on dist-test-slave-mvvj"
I20260808 01:09:04.143270 32593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:09:04.159595 32593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 01:09:04.160876 32593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:09:04.162329 32593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 01:09:04.164912 32593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:09:04.165117 32593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:04.165411 32593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 01:09:04.165586 32593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:04.208356 32593 rpc_server.cc:307] RPC server started. Bound to: 127.31.212.65:40159
I20260808 01:09:04.208544 873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.212.65:40159 every 8 connection(s)
I20260808 01:09:04.213330 32593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 01:09:04.221266 878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:09:04.223693 879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:09:04.227187 882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:09:04.227304 32593 server_base.cc:1034] running on GCE node
I20260808 01:09:04.229893 32593 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:09:04.230189 32593 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:09:04.230396 32593 hybrid_clock.cc:648] HybridClock initialized: now 1786151344230377 us; error 0 us; skew 500 ppm
I20260808 01:09:04.231103 32593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:09:04.233189 874 heartbeater.cc:347] Connected to a master server at 127.31.212.126:36597
I20260808 01:09:04.233601 874 heartbeater.cc:464] Registering TS with master...
I20260808 01:09:04.234037 32593 webserver.cc:460] Webserver started at http://127.31.212.66:42843/ using document root <none> and password file <none>
I20260808 01:09:04.234598 874 heartbeater.cc:511] Master 127.31.212.126:36597 requested a full tablet report, sending...
I20260808 01:09:04.234736 32593 fs_manager.cc:359] Metadata directory not provided
I20260808 01:09:04.234992 32593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:09:04.235347 32593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:09:04.236837 32593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/ts-1-root/instance:
uuid: "3286595befcc479093c91952e8ee51d4"
format_stamp: "Formatted at 2026-08-08 01:09:04 on dist-test-slave-mvvj"
I20260808 01:09:04.237197 743 ts_manager.cc:194] Registered new tserver with Master: 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159)
I20260808 01:09:04.239205 743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45670
I20260808 01:09:04.242321 32593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 01:09:04.246788 887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:04.247609 32593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 01:09:04.247970 32593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/ts-1-root
uuid: "3286595befcc479093c91952e8ee51d4"
format_stamp: "Formatted at 2026-08-08 01:09:04 on dist-test-slave-mvvj"
I20260808 01:09:04.248288 32593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:09:04.268045 32593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 01:09:04.269351 32593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:09:04.270877 32593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 01:09:04.273208 32593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:09:04.273439 32593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:04.273689 32593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 01:09:04.273856 32593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:04.319875 32593 rpc_server.cc:307] RPC server started. Bound to: 127.31.212.66:38485
I20260808 01:09:04.320158 951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.212.66:38485 every 8 connection(s)
I20260808 01:09:04.324743 32593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 01:09:04.332783 956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:09:04.335032 957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:09:04.337167 959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:09:04.337409 32593 server_base.cc:1034] running on GCE node
I20260808 01:09:04.338538 32593 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:09:04.338809 32593 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:09:04.339023 32593 hybrid_clock.cc:648] HybridClock initialized: now 1786151344338998 us; error 0 us; skew 500 ppm
I20260808 01:09:04.339687 32593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:09:04.340091 952 heartbeater.cc:347] Connected to a master server at 127.31.212.126:36597
I20260808 01:09:04.340581 952 heartbeater.cc:464] Registering TS with master...
I20260808 01:09:04.341580 952 heartbeater.cc:511] Master 127.31.212.126:36597 requested a full tablet report, sending...
I20260808 01:09:04.342691 32593 webserver.cc:460] Webserver started at http://127.31.212.67:36001/ using document root <none> and password file <none>
I20260808 01:09:04.343281 32593 fs_manager.cc:359] Metadata directory not provided
I20260808 01:09:04.343518 32593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:09:04.343788 32593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:09:04.344105 743 ts_manager.cc:194] Registered new tserver with Master: 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:04.345345 32593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/ts-2-root/instance:
uuid: "9ed48023032e41438840c4a91055fd51"
format_stamp: "Formatted at 2026-08-08 01:09:04 on dist-test-slave-mvvj"
I20260808 01:09:04.345818 743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45680
I20260808 01:09:04.350472 32593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 01:09:04.354168 965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:04.355101 32593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 01:09:04.355471 32593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/ts-2-root
uuid: "9ed48023032e41438840c4a91055fd51"
format_stamp: "Formatted at 2026-08-08 01:09:04 on dist-test-slave-mvvj"
I20260808 01:09:04.355820 32593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786151308190403-32593-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:09:04.376418 32593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 01:09:04.377825 32593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:09:04.379617 32593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 01:09:04.382309 32593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:09:04.382596 32593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:04.382875 32593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 01:09:04.383097 32593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:04.430644 32593 rpc_server.cc:307] RPC server started. Bound to: 127.31.212.67:32931
I20260808 01:09:04.430831 1031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.212.67:32931 every 8 connection(s)
I20260808 01:09:04.447934 1032 heartbeater.cc:347] Connected to a master server at 127.31.212.126:36597
I20260808 01:09:04.448377 1032 heartbeater.cc:464] Registering TS with master...
I20260808 01:09:04.449133 1032 heartbeater.cc:511] Master 127.31.212.126:36597 requested a full tablet report, sending...
I20260808 01:09:04.451061 743 ts_manager.cc:194] Registered new tserver with Master: 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:04.451448 32593 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017443359s
I20260808 01:09:04.452826 743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45682
I20260808 01:09:04.454442 32593 test_util.cc:268] Using random seed: -999900410
I20260808 01:09:04.483264 743 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45694:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 01:09:04.486582 743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 01:09:04.576081 994 tablet_service.cc:1467] Processing CreateTablet for tablet a8232c0f88b94775baedaf083a8bdd0e (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260808 01:09:04.576701 993 tablet_service.cc:1467] Processing CreateTablet for tablet 019d36b6f9b349058a1b30da6ab047f4 (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260808 01:09:04.577673 994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8232c0f88b94775baedaf083a8bdd0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.579061 992 tablet_service.cc:1467] Processing CreateTablet for tablet a6ec8523a0424a6eb9bdd08d1546e64d (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260808 01:09:04.581171 990 tablet_service.cc:1467] Processing CreateTablet for tablet b38325e90b3d4db983aae4c25685e797 (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260808 01:09:04.583298 989 tablet_service.cc:1467] Processing CreateTablet for tablet b677dbeb972c4f6f83f791244b162490 (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260808 01:09:04.585270 991 tablet_service.cc:1467] Processing CreateTablet for tablet 9158c2015bb449e99d8c2f8bc251c0ad (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260808 01:09:04.585279 836 tablet_service.cc:1467] Processing CreateTablet for tablet 9158c2015bb449e99d8c2f8bc251c0ad (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260808 01:09:04.586566 837 tablet_service.cc:1467] Processing CreateTablet for tablet a6ec8523a0424a6eb9bdd08d1546e64d (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260808 01:09:04.586982 836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9158c2015bb449e99d8c2f8bc251c0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.588160 837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6ec8523a0424a6eb9bdd08d1546e64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.586985 992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6ec8523a0424a6eb9bdd08d1546e64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.588020 834 tablet_service.cc:1467] Processing CreateTablet for tablet b677dbeb972c4f6f83f791244b162490 (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260808 01:09:04.585511 835 tablet_service.cc:1467] Processing CreateTablet for tablet b38325e90b3d4db983aae4c25685e797 (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260808 01:09:04.589982 835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38325e90b3d4db983aae4c25685e797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.590307 993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 019d36b6f9b349058a1b30da6ab047f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.592263 834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b677dbeb972c4f6f83f791244b162490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.593238 838 tablet_service.cc:1467] Processing CreateTablet for tablet 019d36b6f9b349058a1b30da6ab047f4 (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260808 01:09:04.594666 838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 019d36b6f9b349058a1b30da6ab047f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.599402 990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38325e90b3d4db983aae4c25685e797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.601033 989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b677dbeb972c4f6f83f791244b162490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.601312 839 tablet_service.cc:1467] Processing CreateTablet for tablet a8232c0f88b94775baedaf083a8bdd0e (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260808 01:09:04.602691 839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8232c0f88b94775baedaf083a8bdd0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.605189 991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9158c2015bb449e99d8c2f8bc251c0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.626613 915 tablet_service.cc:1467] Processing CreateTablet for tablet a8232c0f88b94775baedaf083a8bdd0e (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260808 01:09:04.629187 914 tablet_service.cc:1467] Processing CreateTablet for tablet 019d36b6f9b349058a1b30da6ab047f4 (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260808 01:09:04.632349 913 tablet_service.cc:1467] Processing CreateTablet for tablet a6ec8523a0424a6eb9bdd08d1546e64d (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260808 01:09:04.637657 911 tablet_service.cc:1467] Processing CreateTablet for tablet b38325e90b3d4db983aae4c25685e797 (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260808 01:09:04.635249 912 tablet_service.cc:1467] Processing CreateTablet for tablet 9158c2015bb449e99d8c2f8bc251c0ad (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260808 01:09:04.641353 910 tablet_service.cc:1467] Processing CreateTablet for tablet b677dbeb972c4f6f83f791244b162490 (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260808 01:09:04.642956 915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8232c0f88b94775baedaf083a8bdd0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.647809 1050 tablet_bootstrap.cc:492] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a: Bootstrap starting.
I20260808 01:09:04.648265 912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9158c2015bb449e99d8c2f8bc251c0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.652294 914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 019d36b6f9b349058a1b30da6ab047f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.655947 911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38325e90b3d4db983aae4c25685e797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.657522 1050 tablet_bootstrap.cc:654] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:04.664413 910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b677dbeb972c4f6f83f791244b162490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.668291 913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6ec8523a0424a6eb9bdd08d1546e64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.669566 1050 tablet_bootstrap.cc:492] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a: No bootstrap required, opened a new log
I20260808 01:09:04.670102 1050 ts_tablet_manager.cc:1397] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.007s
I20260808 01:09:04.673018 1050 raft_consensus.cc:348] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:04.673810 1050 raft_consensus.cc:374] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:04.674117 1050 raft_consensus.cc:729] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Initialized, Role: FOLLOWER
I20260808 01:09:04.675081 1050 consensus_queue.cc:260] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:04.712608 1050 ts_tablet_manager.cc:1428] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent starting tablet: real 0.042s user 0.026s sys 0.009s
I20260808 01:09:04.712234 834 tablet_service.cc:1467] Processing CreateTablet for tablet a48d83231e8e4e94a8b0a92791cb861e (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
W20260808 01:09:04.712788 875 tablet.cc:2369] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:09:04.713819 1050 tablet_bootstrap.cc:492] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Bootstrap starting.
I20260808 01:09:04.713955 834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48d83231e8e4e94a8b0a92791cb861e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.714233 874 heartbeater.cc:503] Master 127.31.212.126:36597 was elected leader, sending a full tablet report...
I20260808 01:09:04.744668 1054 tablet_bootstrap.cc:492] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4: Bootstrap starting.
I20260808 01:09:04.751389 1050 tablet_bootstrap.cc:654] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:04.754418 1051 tablet_bootstrap.cc:492] T a8232c0f88b94775baedaf083a8bdd0e P 9ed48023032e41438840c4a91055fd51: Bootstrap starting.
I20260808 01:09:04.755131 911 tablet_service.cc:1467] Processing CreateTablet for tablet a48d83231e8e4e94a8b0a92791cb861e (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260808 01:09:04.756665 911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48d83231e8e4e94a8b0a92791cb861e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.757936 1054 tablet_bootstrap.cc:654] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:04.765195 1050 tablet_bootstrap.cc:492] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a: No bootstrap required, opened a new log
I20260808 01:09:04.766258 1050 ts_tablet_manager.cc:1397] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent bootstrapping tablet: real 0.053s user 0.013s sys 0.006s
I20260808 01:09:04.770092 1050 raft_consensus.cc:348] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:04.771291 1050 raft_consensus.cc:374] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:04.771579 1050 raft_consensus.cc:729] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Initialized, Role: FOLLOWER
I20260808 01:09:04.772301 1050 consensus_queue.cc:260] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:04.775063 1054 tablet_bootstrap.cc:492] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4: No bootstrap required, opened a new log
I20260808 01:09:04.775534 1054 ts_tablet_manager.cc:1397] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4: Time spent bootstrapping tablet: real 0.031s user 0.019s sys 0.002s
I20260808 01:09:04.778465 1054 raft_consensus.cc:348] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:04.778499 994 tablet_service.cc:1467] Processing CreateTablet for tablet a48d83231e8e4e94a8b0a92791cb861e (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260808 01:09:04.779405 1054 raft_consensus.cc:374] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:04.780300 994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48d83231e8e4e94a8b0a92791cb861e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.781700 1050 ts_tablet_manager.cc:1428] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260808 01:09:04.782627 1050 tablet_bootstrap.cc:492] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Bootstrap starting.
I20260808 01:09:04.788074 837 tablet_service.cc:1467] Processing CreateTablet for tablet 3c615b0358fb4ffd8dd027ce0d401f2d (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260808 01:09:04.789161 1050 tablet_bootstrap.cc:654] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:04.789459 837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c615b0358fb4ffd8dd027ce0d401f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.790299 1051 tablet_bootstrap.cc:654] T a8232c0f88b94775baedaf083a8bdd0e P 9ed48023032e41438840c4a91055fd51: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:04.779728 1054 raft_consensus.cc:729] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Initialized, Role: FOLLOWER
I20260808 01:09:04.792441 1054 consensus_queue.cc:260] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:04.798900 1051 tablet_bootstrap.cc:492] T a8232c0f88b94775baedaf083a8bdd0e P 9ed48023032e41438840c4a91055fd51: No bootstrap required, opened a new log
I20260808 01:09:04.799618 1051 ts_tablet_manager.cc:1397] T a8232c0f88b94775baedaf083a8bdd0e P 9ed48023032e41438840c4a91055fd51: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.000s
I20260808 01:09:04.802613 991 tablet_service.cc:1467] Processing CreateTablet for tablet 3c615b0358fb4ffd8dd027ce0d401f2d (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260808 01:09:04.798583 912 tablet_service.cc:1467] Processing CreateTablet for tablet 3c615b0358fb4ffd8dd027ce0d401f2d (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260808 01:09:04.804227 991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c615b0358fb4ffd8dd027ce0d401f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.808681 912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c615b0358fb4ffd8dd027ce0d401f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.803473 1051 raft_consensus.cc:348] T a8232c0f88b94775baedaf083a8bdd0e P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } }
I20260808 01:09:04.811995 1051 raft_consensus.cc:374] T a8232c0f88b94775baedaf083a8bdd0e P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:04.812424 1051 raft_consensus.cc:729] T a8232c0f88b94775baedaf083a8bdd0e P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Initialized, Role: FOLLOWER
I20260808 01:09:04.815001 834 tablet_service.cc:1467] Processing CreateTablet for tablet c52df16f25154c81adfb9d5b50d8bf01 (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260808 01:09:04.815580 1051 consensus_queue.cc:260] T a8232c0f88b94775baedaf083a8bdd0e P 9ed48023032e41438840c4a91055fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } }
I20260808 01:09:04.816529 834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52df16f25154c81adfb9d5b50d8bf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.822728 911 tablet_service.cc:1467] Processing CreateTablet for tablet c52df16f25154c81adfb9d5b50d8bf01 (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260808 01:09:04.824191 838 tablet_service.cc:1467] Processing CreateTablet for tablet 863fef312d0e4f6393c520f611b6af00 (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260808 01:09:04.825204 911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52df16f25154c81adfb9d5b50d8bf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.825585 838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863fef312d0e4f6393c520f611b6af00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.826195 990 tablet_service.cc:1467] Processing CreateTablet for tablet c52df16f25154c81adfb9d5b50d8bf01 (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260808 01:09:04.828044 990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52df16f25154c81adfb9d5b50d8bf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.838153 913 tablet_service.cc:1467] Processing CreateTablet for tablet 863fef312d0e4f6393c520f611b6af00 (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260808 01:09:04.839850 913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863fef312d0e4f6393c520f611b6af00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.851018 994 tablet_service.cc:1467] Processing CreateTablet for tablet 863fef312d0e4f6393c520f611b6af00 (DEFAULT_TABLE table=test-workload [id=595c3673ed984c278ccec1cecb158125]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260808 01:09:04.852528 994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863fef312d0e4f6393c520f611b6af00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:04.855468 1053 raft_consensus.cc:482] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:04.856030 1053 raft_consensus.cc:504] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:04.858567 1053 leader_election.cc:288] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:04.873207 1032 heartbeater.cc:503] Master 127.31.212.126:36597 was elected leader, sending a full tablet report...
I20260808 01:09:04.885653 1062 raft_consensus.cc:482] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:04.885706 952 heartbeater.cc:503] Master 127.31.212.126:36597 was elected leader, sending a full tablet report...
I20260808 01:09:04.888527 1062 raft_consensus.cc:504] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:04.898911 1051 ts_tablet_manager.cc:1428] T a8232c0f88b94775baedaf083a8bdd0e P 9ed48023032e41438840c4a91055fd51: Time spent starting tablet: real 0.099s user 0.011s sys 0.008s
I20260808 01:09:04.907647 1054 ts_tablet_manager.cc:1428] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4: Time spent starting tablet: real 0.132s user 0.017s sys 0.018s
I20260808 01:09:04.908782 1054 tablet_bootstrap.cc:492] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4: Bootstrap starting.
I20260808 01:09:04.911993 1050 tablet_bootstrap.cc:492] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a: No bootstrap required, opened a new log
I20260808 01:09:04.912590 1050 ts_tablet_manager.cc:1397] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent bootstrapping tablet: real 0.130s user 0.047s sys 0.059s
I20260808 01:09:04.914966 1051 tablet_bootstrap.cc:492] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51: Bootstrap starting.
I20260808 01:09:04.916162 1050 raft_consensus.cc:348] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:04.917047 1050 raft_consensus.cc:374] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:04.917462 1050 raft_consensus.cc:729] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Initialized, Role: FOLLOWER
I20260808 01:09:04.918453 1050 consensus_queue.cc:260] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:04.923969 1050 ts_tablet_manager.cc:1428] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260808 01:09:04.929596 1050 tablet_bootstrap.cc:492] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a: Bootstrap starting.
I20260808 01:09:04.933862 1062 leader_election.cc:288] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:04.927942 1054 tablet_bootstrap.cc:654] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:04.924852 1051 tablet_bootstrap.cc:654] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:04.942227 1050 tablet_bootstrap.cc:654] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:04.942323 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
W20260808 01:09:04.944486 812 leader_election.cc:341] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:04.949982 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
W20260808 01:09:04.953868 812 leader_election.cc:341] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:04.954598 812 leader_election.cc:302] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51
I20260808 01:09:04.956212 1053 raft_consensus.cc:2726] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:04.957152 1051 tablet_bootstrap.cc:492] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51: No bootstrap required, opened a new log
I20260808 01:09:04.960152 1051 ts_tablet_manager.cc:1397] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51: Time spent bootstrapping tablet: real 0.045s user 0.015s sys 0.004s
I20260808 01:09:04.964157 1051 raft_consensus.cc:348] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:04.965233 1051 raft_consensus.cc:374] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:04.965600 1051 raft_consensus.cc:729] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Initialized, Role: FOLLOWER
I20260808 01:09:04.966594 1051 consensus_queue.cc:260] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:04.968613 849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ec8523a0424a6eb9bdd08d1546e64d" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
W20260808 01:09:04.971019 890 leader_election.cc:341] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:04.975075 1050 tablet_bootstrap.cc:492] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a: No bootstrap required, opened a new log
I20260808 01:09:04.975649 1050 ts_tablet_manager.cc:1397] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent bootstrapping tablet: real 0.046s user 0.019s sys 0.010s
I20260808 01:09:04.976046 1054 tablet_bootstrap.cc:492] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4: No bootstrap required, opened a new log
I20260808 01:09:04.976589 1054 ts_tablet_manager.cc:1397] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4: Time spent bootstrapping tablet: real 0.068s user 0.018s sys 0.019s
I20260808 01:09:04.978142 1051 ts_tablet_manager.cc:1428] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51: Time spent starting tablet: real 0.018s user 0.004s sys 0.004s
I20260808 01:09:04.978535 1050 raft_consensus.cc:348] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } }
I20260808 01:09:04.979353 1050 raft_consensus.cc:374] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:04.979650 1050 raft_consensus.cc:729] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Initialized, Role: FOLLOWER
I20260808 01:09:04.979677 1054 raft_consensus.cc:348] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:04.980746 1054 raft_consensus.cc:374] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:04.980544 1050 consensus_queue.cc:260] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } }
I20260808 01:09:04.981122 1054 raft_consensus.cc:729] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Initialized, Role: FOLLOWER
I20260808 01:09:04.982074 1054 consensus_queue.cc:260] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:04.982959 1050 ts_tablet_manager.cc:1428] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260808 01:09:04.984159 1050 tablet_bootstrap.cc:492] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a: Bootstrap starting.
I20260808 01:09:04.986016 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ec8523a0424a6eb9bdd08d1546e64d" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
W20260808 01:09:04.987747 888 leader_election.cc:341] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:04.988305 888 leader_election.cc:302] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3286595befcc479093c91952e8ee51d4; no voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51
I20260808 01:09:04.989230 1062 raft_consensus.cc:2726] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:04.990483 1050 tablet_bootstrap.cc:654] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:04.990625 1054 ts_tablet_manager.cc:1428] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4: Time spent starting tablet: real 0.014s user 0.005s sys 0.002s
I20260808 01:09:04.991667 1054 tablet_bootstrap.cc:492] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4: Bootstrap starting.
I20260808 01:09:04.996824 1050 tablet_bootstrap.cc:492] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a: No bootstrap required, opened a new log
I20260808 01:09:04.997306 1050 ts_tablet_manager.cc:1397] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260808 01:09:04.998286 1054 tablet_bootstrap.cc:654] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:04.988585 1051 tablet_bootstrap.cc:492] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51: Bootstrap starting.
I20260808 01:09:04.999715 1050 raft_consensus.cc:348] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.000548 1050 raft_consensus.cc:374] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.000840 1050 raft_consensus.cc:729] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.001566 1050 consensus_queue.cc:260] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.004395 1050 ts_tablet_manager.cc:1428] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 01:09:05.005530 1050 tablet_bootstrap.cc:492] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Bootstrap starting.
I20260808 01:09:05.006031 1051 tablet_bootstrap.cc:654] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.013074 1050 tablet_bootstrap.cc:654] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.018333 1051 tablet_bootstrap.cc:492] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51: No bootstrap required, opened a new log
I20260808 01:09:05.018888 1051 ts_tablet_manager.cc:1397] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.009s
I20260808 01:09:05.019974 1054 tablet_bootstrap.cc:492] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4: No bootstrap required, opened a new log
I20260808 01:09:05.020741 1054 ts_tablet_manager.cc:1397] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.018s
I20260808 01:09:05.021857 1051 raft_consensus.cc:348] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.022722 1051 raft_consensus.cc:374] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.022992 1051 raft_consensus.cc:729] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.024143 1050 tablet_bootstrap.cc:492] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a: No bootstrap required, opened a new log
I20260808 01:09:05.023635 1051 consensus_queue.cc:260] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.023593 1054 raft_consensus.cc:348] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.024637 1054 raft_consensus.cc:374] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.024641 1050 ts_tablet_manager.cc:1397] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.000s
I20260808 01:09:05.025234 1054 raft_consensus.cc:729] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.026183 1051 ts_tablet_manager.cc:1428] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 01:09:05.026268 1054 consensus_queue.cc:260] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.027864 1050 raft_consensus.cc:348] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.028600 1050 raft_consensus.cc:374] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.028927 1054 ts_tablet_manager.cc:1428] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260808 01:09:05.028895 1050 raft_consensus.cc:729] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.029850 1050 consensus_queue.cc:260] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.032943 1054 tablet_bootstrap.cc:492] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4: Bootstrap starting.
I20260808 01:09:05.035228 1050 ts_tablet_manager.cc:1428] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260808 01:09:05.036374 1050 tablet_bootstrap.cc:492] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a: Bootstrap starting.
I20260808 01:09:05.027251 1051 tablet_bootstrap.cc:492] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51: Bootstrap starting.
W20260808 01:09:05.041208 795 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:09:05.042129 1054 tablet_bootstrap.cc:654] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.042896 1050 tablet_bootstrap.cc:654] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.047324 1053 raft_consensus.cc:482] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:05.047871 1053 raft_consensus.cc:504] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } }
I20260808 01:09:05.049760 1053 leader_election.cc:288] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:05.051370 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8232c0f88b94775baedaf083a8bdd0e" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:05.052366 1005 raft_consensus.cc:2454] T a8232c0f88b94775baedaf083a8bdd0e P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 0.
I20260808 01:09:05.054368 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8232c0f88b94775baedaf083a8bdd0e" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:05.055135 1054 tablet_bootstrap.cc:492] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4: No bootstrap required, opened a new log
I20260808 01:09:05.055673 1054 ts_tablet_manager.cc:1397] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.011s
W20260808 01:09:05.056437 812 leader_election.cc:341] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:05.057163 1050 tablet_bootstrap.cc:492] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a: No bootstrap required, opened a new log
I20260808 01:09:05.057195 812 leader_election.cc:302] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters: 3286595befcc479093c91952e8ee51d4
I20260808 01:09:05.054365 1051 tablet_bootstrap.cc:654] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.057715 1050 ts_tablet_manager.cc:1397] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.004s
I20260808 01:09:05.058280 1053 raft_consensus.cc:2781] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:05.058828 1053 raft_consensus.cc:482] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:05.059198 1053 raft_consensus.cc:3037] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.058246 1054 raft_consensus.cc:348] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } }
I20260808 01:09:05.060740 1054 raft_consensus.cc:374] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.061108 1054 raft_consensus.cc:729] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.061507 1050 raft_consensus.cc:348] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.062059 1054 consensus_queue.cc:260] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } }
I20260808 01:09:05.062569 1050 raft_consensus.cc:374] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.063155 1050 raft_consensus.cc:729] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.064313 1050 consensus_queue.cc:260] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.066854 1054 ts_tablet_manager.cc:1428] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4: Time spent starting tablet: real 0.011s user 0.003s sys 0.002s
I20260808 01:09:05.067992 1054 tablet_bootstrap.cc:492] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4: Bootstrap starting.
I20260808 01:09:05.074227 1054 tablet_bootstrap.cc:654] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.078913 1053 raft_consensus.cc:504] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } }
I20260808 01:09:05.083077 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8232c0f88b94775baedaf083a8bdd0e" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:05.083889 1005 raft_consensus.cc:3037] T a8232c0f88b94775baedaf083a8bdd0e P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.086525 1054 tablet_bootstrap.cc:492] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4: No bootstrap required, opened a new log
I20260808 01:09:05.087113 1054 ts_tablet_manager.cc:1397] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.009s
I20260808 01:09:05.090133 1054 raft_consensus.cc:348] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.090960 1005 raft_consensus.cc:2454] T a8232c0f88b94775baedaf083a8bdd0e P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 1.
I20260808 01:09:05.091130 1054 raft_consensus.cc:374] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.091537 1054 raft_consensus.cc:729] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.092250 812 leader_election.cc:302] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:05.092489 1054 consensus_queue.cc:260] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.097915 1054 ts_tablet_manager.cc:1428] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s
I20260808 01:09:05.099028 1054 tablet_bootstrap.cc:492] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4: Bootstrap starting.
I20260808 01:09:05.104755 1053 leader_election.cc:288] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 election: Requested vote from peers 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:05.105319 1054 tablet_bootstrap.cc:654] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.116268 1053 raft_consensus.cc:2781] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:05.116791 1053 raft_consensus.cc:686] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Becoming Leader. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:05.117812 1054 tablet_bootstrap.cc:492] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4: No bootstrap required, opened a new log
I20260808 01:09:05.117623 1053 consensus_queue.cc:237] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } }
I20260808 01:09:05.118476 1054 ts_tablet_manager.cc:1397] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.009s
I20260808 01:09:05.123445 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8232c0f88b94775baedaf083a8bdd0e" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:05.124119 925 raft_consensus.cc:3037] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.125926 1054 raft_consensus.cc:348] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.126806 1054 raft_consensus.cc:374] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.127161 1054 raft_consensus.cc:729] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.128077 1054 consensus_queue.cc:260] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.130856 1054 ts_tablet_manager.cc:1428] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20260808 01:09:05.131949 1054 tablet_bootstrap.cc:492] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4: Bootstrap starting.
I20260808 01:09:05.138065 1054 tablet_bootstrap.cc:654] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.142937 925 raft_consensus.cc:2454] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 1.
I20260808 01:09:05.149513 1080 raft_consensus.cc:482] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:05.150193 1080 raft_consensus.cc:504] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.152197 1080 leader_election.cc:288] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.152428 1054 tablet_bootstrap.cc:492] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4: No bootstrap required, opened a new log
I20260808 01:09:05.153108 1054 ts_tablet_manager.cc:1397] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.008s
I20260808 01:09:05.154733 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
W20260808 01:09:05.156486 812 leader_election.cc:341] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:05.156553 1054 raft_consensus.cc:348] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.157469 1054 raft_consensus.cc:374] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.153204 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:05.157825 1054 raft_consensus.cc:729] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.158203 1051 tablet_bootstrap.cc:492] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51: No bootstrap required, opened a new log
I20260808 01:09:05.158957 1051 ts_tablet_manager.cc:1397] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51: Time spent bootstrapping tablet: real 0.132s user 0.033s sys 0.069s
I20260808 01:09:05.158695 1054 consensus_queue.cc:260] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.161360 1050 ts_tablet_manager.cc:1428] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent starting tablet: real 0.103s user 0.049s sys 0.034s
I20260808 01:09:05.161863 925 raft_consensus.cc:2454] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 0.
I20260808 01:09:05.162388 1050 tablet_bootstrap.cc:492] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a: Bootstrap starting.
I20260808 01:09:05.162110 1051 raft_consensus.cc:348] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.162899 1051 raft_consensus.cc:374] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.163225 1051 raft_consensus.cc:729] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.164103 1051 consensus_queue.cc:260] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.166167 1054 ts_tablet_manager.cc:1428] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4: Time spent starting tablet: real 0.013s user 0.005s sys 0.003s
I20260808 01:09:05.167224 1054 tablet_bootstrap.cc:492] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4: Bootstrap starting.
I20260808 01:09:05.169131 1050 tablet_bootstrap.cc:654] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.171056 1051 ts_tablet_manager.cc:1428] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260808 01:09:05.173875 1054 tablet_bootstrap.cc:654] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.180299 1051 tablet_bootstrap.cc:492] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51: Bootstrap starting.
I20260808 01:09:05.186182 812 leader_election.cc:302] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters: 9ed48023032e41438840c4a91055fd51
I20260808 01:09:05.187103 1053 raft_consensus.cc:2781] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:05.187476 1053 raft_consensus.cc:482] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:05.187793 1054 tablet_bootstrap.cc:492] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4: No bootstrap required, opened a new log
I20260808 01:09:05.187815 1053 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.186972 742 catalog_manager.cc:5495] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a reported cstate change: term changed from 0 to 1, leader changed from <none> to 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65). New cstate: current_term: 1 leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:05.188486 1054 ts_tablet_manager.cc:1397] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.004s
I20260808 01:09:05.190325 1051 tablet_bootstrap.cc:654] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.191555 1054 raft_consensus.cc:348] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.192305 1054 raft_consensus.cc:374] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.192601 1054 raft_consensus.cc:729] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.193487 1054 consensus_queue.cc:260] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.196223 1050 tablet_bootstrap.cc:492] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a: No bootstrap required, opened a new log
I20260808 01:09:05.196735 1050 ts_tablet_manager.cc:1397] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent bootstrapping tablet: real 0.035s user 0.015s sys 0.009s
I20260808 01:09:05.196641 1053 raft_consensus.cc:504] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.199344 1053 leader_election.cc:288] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 election: Requested vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.199615 1050 raft_consensus.cc:348] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.200366 1050 raft_consensus.cc:374] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.200647 1050 raft_consensus.cc:729] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.201282 1050 consensus_queue.cc:260] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.201742 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:05.202441 925 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.203317 1054 ts_tablet_manager.cc:1428] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4: Time spent starting tablet: real 0.014s user 0.004s sys 0.001s
I20260808 01:09:05.203428 1050 ts_tablet_manager.cc:1428] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 01:09:05.204304 1050 tablet_bootstrap.cc:492] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Bootstrap starting.
I20260808 01:09:05.208437 1078 raft_consensus.cc:482] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:05.209139 1078 raft_consensus.cc:504] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.211668 1054 tablet_bootstrap.cc:492] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4: Bootstrap starting.
I20260808 01:09:05.211970 1078 leader_election.cc:288] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.212692 1050 tablet_bootstrap.cc:654] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.216980 1051 tablet_bootstrap.cc:492] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51: No bootstrap required, opened a new log
I20260808 01:09:05.217512 1051 ts_tablet_manager.cc:1397] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.002s
I20260808 01:09:05.217351 925 raft_consensus.cc:2454] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 1.
I20260808 01:09:05.220152 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:05.220546 1054 tablet_bootstrap.cc:654] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.220486 1051 raft_consensus.cc:348] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.221343 1051 raft_consensus.cc:374] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.221676 1051 raft_consensus.cc:729] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.223191 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ec8523a0424a6eb9bdd08d1546e64d" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:05.223793 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ec8523a0424a6eb9bdd08d1546e64d" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:05.223985 924 raft_consensus.cc:2454] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 0.
W20260808 01:09:05.225009 812 leader_election.cc:341] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:05.225119 1005 raft_consensus.cc:2454] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 0.
I20260808 01:09:05.225961 812 leader_election.cc:302] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters: 9ed48023032e41438840c4a91055fd51
I20260808 01:09:05.226750 1078 raft_consensus.cc:2781] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:05.226511 1051 consensus_queue.cc:260] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.226881 812 leader_election.cc:302] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters:
I20260808 01:09:05.227658 1078 raft_consensus.cc:686] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Becoming Leader. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:05.231117 1078 consensus_queue.cc:237] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.232721 1050 tablet_bootstrap.cc:492] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a: No bootstrap required, opened a new log
I20260808 01:09:05.233189 1050 ts_tablet_manager.cc:1397] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.001s
I20260808 01:09:05.234028 1054 tablet_bootstrap.cc:492] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4: No bootstrap required, opened a new log
I20260808 01:09:05.234453 1054 ts_tablet_manager.cc:1397] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.004s
I20260808 01:09:05.227819 1080 raft_consensus.cc:2781] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:05.235126 1080 raft_consensus.cc:482] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:05.235464 1080 raft_consensus.cc:3037] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.235679 1050 raft_consensus.cc:348] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.236650 1050 raft_consensus.cc:374] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.237032 1050 raft_consensus.cc:729] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.238128 1050 consensus_queue.cc:260] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.240898 1050 ts_tablet_manager.cc:1428] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 01:09:05.241185 1054 raft_consensus.cc:348] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.241803 1050 tablet_bootstrap.cc:492] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Bootstrap starting.
I20260808 01:09:05.241868 1054 raft_consensus.cc:374] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.242209 1054 raft_consensus.cc:729] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.244819 1054 consensus_queue.cc:260] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.230058 1051 ts_tablet_manager.cc:1428] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51: Time spent starting tablet: real 0.012s user 0.005s sys 0.004s
I20260808 01:09:05.247900 1051 tablet_bootstrap.cc:492] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51: Bootstrap starting.
I20260808 01:09:05.248620 1050 tablet_bootstrap.cc:654] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.254583 1051 tablet_bootstrap.cc:654] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.258639 1080 raft_consensus.cc:504] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.261128 1050 tablet_bootstrap.cc:492] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a: No bootstrap required, opened a new log
I20260808 01:09:05.261041 1054 ts_tablet_manager.cc:1428] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4: Time spent starting tablet: real 0.026s user 0.004s sys 0.003s
I20260808 01:09:05.261698 1050 ts_tablet_manager.cc:1397] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.005s
I20260808 01:09:05.262553 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ec8523a0424a6eb9bdd08d1546e64d" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:05.263335 925 raft_consensus.cc:3037] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.264233 1050 raft_consensus.cc:348] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.265056 1050 raft_consensus.cc:374] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.265319 1050 raft_consensus.cc:729] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.266192 1050 consensus_queue.cc:260] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.267452 1054 tablet_bootstrap.cc:492] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4: Bootstrap starting.
I20260808 01:09:05.269309 1050 ts_tablet_manager.cc:1428] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 01:09:05.272218 1051 tablet_bootstrap.cc:492] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51: No bootstrap required, opened a new log
I20260808 01:09:05.272702 1051 ts_tablet_manager.cc:1397] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.006s
I20260808 01:09:05.273499 1054 tablet_bootstrap.cc:654] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.275071 1080 leader_election.cc:288] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 election: Requested vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.274974 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ec8523a0424a6eb9bdd08d1546e64d" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:05.275919 1005 raft_consensus.cc:3037] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.279028 1051 raft_consensus.cc:348] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.279976 1051 raft_consensus.cc:374] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.280064 925 raft_consensus.cc:2454] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 1.
I20260808 01:09:05.280308 1051 raft_consensus.cc:729] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.281450 1051 consensus_queue.cc:260] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.285154 1051 ts_tablet_manager.cc:1428] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260808 01:09:05.285660 812 leader_election.cc:302] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters:
I20260808 01:09:05.286468 1005 raft_consensus.cc:2454] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 1.
I20260808 01:09:05.286585 1051 tablet_bootstrap.cc:492] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51: Bootstrap starting.
I20260808 01:09:05.290086 1054 tablet_bootstrap.cc:492] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4: No bootstrap required, opened a new log
I20260808 01:09:05.290668 1054 ts_tablet_manager.cc:1397] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.001s
I20260808 01:09:05.291109 1080 raft_consensus.cc:2781] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:05.293725 1051 tablet_bootstrap.cc:654] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.293291 1054 raft_consensus.cc:348] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.296001 1054 raft_consensus.cc:374] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.296141 1080 raft_consensus.cc:686] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Becoming Leader. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:05.296402 1054 raft_consensus.cc:729] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.297075 1080 consensus_queue.cc:237] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.297114 1054 consensus_queue.cc:260] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.299296 1054 ts_tablet_manager.cc:1428] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260808 01:09:05.294543 743 catalog_manager.cc:5495] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a reported cstate change: term changed from 0 to 1, leader changed from <none> to 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65). New cstate: current_term: 1 leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:05.311157 1051 tablet_bootstrap.cc:492] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51: No bootstrap required, opened a new log
I20260808 01:09:05.311769 1051 ts_tablet_manager.cc:1397] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.005s
I20260808 01:09:05.314841 1051 raft_consensus.cc:348] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.315758 1051 raft_consensus.cc:374] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.316136 1051 raft_consensus.cc:729] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.317019 1051 consensus_queue.cc:260] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.317807 1080 raft_consensus.cc:482] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:05.318347 1080 raft_consensus.cc:504] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.320152 1060 raft_consensus.cc:482] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:05.320680 1080 leader_election.cc:288] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.320675 1060 raft_consensus.cc:504] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.321329 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:05.322031 925 raft_consensus.cc:2454] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 0.
I20260808 01:09:05.323344 812 leader_election.cc:302] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters:
I20260808 01:09:05.323226 741 catalog_manager.cc:5495] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a reported cstate change: term changed from 0 to 1, leader changed from <none> to 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65). New cstate: current_term: 1 leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:05.324772 1080 raft_consensus.cc:2781] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:05.328372 1080 raft_consensus.cc:482] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:05.324909 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:05.328774 1080 raft_consensus.cc:3037] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.329037 1005 raft_consensus.cc:2454] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 0.
I20260808 01:09:05.333737 1060 leader_election.cc:288] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:05.337917 1051 ts_tablet_manager.cc:1428] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20260808 01:09:05.340194 1051 tablet_bootstrap.cc:492] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51: Bootstrap starting.
I20260808 01:09:05.341127 1080 raft_consensus.cc:504] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.344013 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:05.344693 925 raft_consensus.cc:3037] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.349582 1051 tablet_bootstrap.cc:654] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.352488 1080 leader_election.cc:288] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 election: Requested vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.353821 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:05.353904 925 raft_consensus.cc:2454] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 1.
I20260808 01:09:05.354601 1005 raft_consensus.cc:3037] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.355537 812 leader_election.cc:302] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters:
I20260808 01:09:05.357079 1060 raft_consensus.cc:482] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:05.357142 1080 raft_consensus.cc:2781] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:05.357718 1060 raft_consensus.cc:504] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.358266 1080 raft_consensus.cc:686] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Becoming Leader. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:05.360169 1060 leader_election.cc:288] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:05.359541 1080 consensus_queue.cc:237] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.364622 849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:05.364789 848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:05.365633 849 raft_consensus.cc:2454] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 0.
I20260808 01:09:05.365600 1005 raft_consensus.cc:2454] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 1.
I20260808 01:09:05.368057 968 leader_election.cc:302] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:05.369294 1060 raft_consensus.cc:2781] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:05.369740 1060 raft_consensus.cc:482] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:05.370133 1060 raft_consensus.cc:3037] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.370919 1078 raft_consensus.cc:482] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:05.371372 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:05.371574 1078 raft_consensus.cc:504] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.372540 924 raft_consensus.cc:2379] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ed48023032e41438840c4a91055fd51 in current term 1: Already voted for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in this term.
I20260808 01:09:05.373966 966 leader_election.cc:302] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ed48023032e41438840c4a91055fd51; no voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a
I20260808 01:09:05.374398 1078 leader_election.cc:288] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.375650 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:05.376389 925 raft_consensus.cc:2454] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 0.
I20260808 01:09:05.377519 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48d83231e8e4e94a8b0a92791cb861e" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:05.378523 924 raft_consensus.cc:2454] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 0.
I20260808 01:09:05.378928 1060 raft_consensus.cc:504] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.380038 812 leader_election.cc:302] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters:
I20260808 01:09:05.381099 1060 leader_election.cc:288] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 1 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:05.381784 1060 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.383591 849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:05.384966 849 raft_consensus.cc:3037] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.385999 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48d83231e8e4e94a8b0a92791cb861e" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:05.389192 1078 raft_consensus.cc:2781] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:05.389846 1051 tablet_bootstrap.cc:492] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51: No bootstrap required, opened a new log
I20260808 01:09:05.391286 1005 raft_consensus.cc:2454] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 0.
I20260808 01:09:05.390409 1078 raft_consensus.cc:482] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:05.392217 1078 raft_consensus.cc:3037] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.397436 849 raft_consensus.cc:2454] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 1.
I20260808 01:09:05.398967 968 leader_election.cc:302] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:05.399446 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:05.397876 1060 raft_consensus.cc:2726] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:05.399523 1051 ts_tablet_manager.cc:1397] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51: Time spent bootstrapping tablet: real 0.060s user 0.024s sys 0.023s
I20260808 01:09:05.400383 1102 raft_consensus.cc:2781] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:05.400914 1102 raft_consensus.cc:686] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 1 LEADER]: Becoming Leader. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Running, Role: LEADER
I20260808 01:09:05.401612 924 raft_consensus.cc:3037] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.403724 1102 consensus_queue.cc:237] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.407106 741 catalog_manager.cc:5495] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a reported cstate change: term changed from 0 to 1, leader changed from <none> to 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65). New cstate: current_term: 1 leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:05.408010 1051 raft_consensus.cc:348] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.408962 1051 raft_consensus.cc:374] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.409337 1051 raft_consensus.cc:729] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.410202 1051 consensus_queue.cc:260] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.410837 924 raft_consensus.cc:2454] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 1.
I20260808 01:09:05.412772 1051 ts_tablet_manager.cc:1428] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260808 01:09:05.413733 1051 tablet_bootstrap.cc:492] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51: Bootstrap starting.
I20260808 01:09:05.420534 1051 tablet_bootstrap.cc:654] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.420274 1078 raft_consensus.cc:504] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.424604 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48d83231e8e4e94a8b0a92791cb861e" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:05.424858 1078 leader_election.cc:288] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 election: Requested vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.425487 924 raft_consensus.cc:3037] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.425673 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48d83231e8e4e94a8b0a92791cb861e" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:05.426331 1005 raft_consensus.cc:3037] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.430831 1051 tablet_bootstrap.cc:492] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51: No bootstrap required, opened a new log
I20260808 01:09:05.431375 1051 ts_tablet_manager.cc:1397] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20260808 01:09:05.432983 1005 raft_consensus.cc:2454] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 1.
I20260808 01:09:05.431564 741 catalog_manager.cc:5495] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ed48023032e41438840c4a91055fd51 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "9ed48023032e41438840c4a91055fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:05.434216 1051 raft_consensus.cc:348] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.434433 812 leader_election.cc:302] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:05.435272 1051 raft_consensus.cc:374] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.435284 1078 raft_consensus.cc:2781] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:05.435782 1051 raft_consensus.cc:729] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.436219 1078 raft_consensus.cc:686] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Becoming Leader. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:05.436882 1080 raft_consensus.cc:482] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:05.437366 1080 raft_consensus.cc:504] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.437654 1078 consensus_queue.cc:237] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.439466 1080 leader_election.cc:288] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.440593 924 raft_consensus.cc:2454] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 1.
I20260808 01:09:05.441486 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863fef312d0e4f6393c520f611b6af00" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:05.442641 1051 consensus_queue.cc:260] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.443727 1005 raft_consensus.cc:2454] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 0.
I20260808 01:09:05.444953 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863fef312d0e4f6393c520f611b6af00" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:05.446026 812 leader_election.cc:302] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:05.446981 1080 raft_consensus.cc:2781] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:05.447331 1080 raft_consensus.cc:482] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:05.448050 1080 raft_consensus.cc:3037] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.448432 1051 ts_tablet_manager.cc:1428] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51: Time spent starting tablet: real 0.017s user 0.009s sys 0.000s
I20260808 01:09:05.449648 1051 tablet_bootstrap.cc:492] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51: Bootstrap starting.
I20260808 01:09:05.445745 924 raft_consensus.cc:2454] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 0.
I20260808 01:09:05.456521 1051 tablet_bootstrap.cc:654] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:05.459460 1080 raft_consensus.cc:504] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.463626 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863fef312d0e4f6393c520f611b6af00" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:05.464519 924 raft_consensus.cc:3037] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.474491 1080 leader_election.cc:288] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 election: Requested vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.474725 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863fef312d0e4f6393c520f611b6af00" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:05.475314 1005 raft_consensus.cc:3037] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.480150 924 raft_consensus.cc:2454] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 1.
I20260808 01:09:05.481317 743 catalog_manager.cc:5495] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a reported cstate change: term changed from 0 to 1, leader changed from <none> to 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65). New cstate: current_term: 1 leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:05.482301 1005 raft_consensus.cc:2454] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 1.
I20260808 01:09:05.481339 812 leader_election.cc:302] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters:
I20260808 01:09:05.485287 1051 tablet_bootstrap.cc:492] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51: No bootstrap required, opened a new log
I20260808 01:09:05.485738 1051 ts_tablet_manager.cc:1397] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.015s
I20260808 01:09:05.488219 1051 raft_consensus.cc:348] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.489015 1051 raft_consensus.cc:374] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:05.489305 1051 raft_consensus.cc:729] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Initialized, Role: FOLLOWER
I20260808 01:09:05.490378 1080 raft_consensus.cc:2781] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:05.489970 1051 consensus_queue.cc:260] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.490880 1080 raft_consensus.cc:686] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Becoming Leader. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:05.491745 1062 raft_consensus.cc:482] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:05.491786 1080 consensus_queue.cc:237] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.492408 1062 raft_consensus.cc:504] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.492961 1051 ts_tablet_manager.cc:1428] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 01:09:05.494477 1062 leader_election.cc:288] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.495407 849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:05.496126 849 raft_consensus.cc:2454] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 0.
I20260808 01:09:05.497305 890 leader_election.cc:302] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters:
I20260808 01:09:05.498353 1062 raft_consensus.cc:2781] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:05.498724 1062 raft_consensus.cc:482] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:05.499013 1062 raft_consensus.cc:3037] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.503525 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:05.504292 1005 raft_consensus.cc:2454] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 0.
I20260808 01:09:05.506929 741 catalog_manager.cc:5495] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a reported cstate change: term changed from 0 to 1, leader changed from <none> to 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65). New cstate: current_term: 1 leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:05.509305 1062 raft_consensus.cc:504] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.513538 1062 leader_election.cc:288] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 1 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.514207 849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:05.514204 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:05.515051 1005 raft_consensus.cc:3037] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.515022 849 raft_consensus.cc:3037] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.522665 849 raft_consensus.cc:2454] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 1.
I20260808 01:09:05.522665 1005 raft_consensus.cc:2454] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 1.
I20260808 01:09:05.524226 890 leader_election.cc:302] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters:
I20260808 01:09:05.525130 1062 raft_consensus.cc:2781] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:05.526146 1062 raft_consensus.cc:686] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 1 LEADER]: Becoming Leader. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Running, Role: LEADER
I20260808 01:09:05.526978 1062 consensus_queue.cc:237] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.531499 1080 consensus_queue.cc:1035] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:05.535405 741 catalog_manager.cc:5495] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3286595befcc479093c91952e8ee51d4 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "3286595befcc479093c91952e8ee51d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:05.536774 1080 raft_consensus.cc:482] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:05.537364 1080 raft_consensus.cc:504] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.539705 1080 leader_election.cc:288] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.540652 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:05.540867 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:05.541357 925 raft_consensus.cc:2454] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 0.
I20260808 01:09:05.541569 1005 raft_consensus.cc:2454] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 0.
I20260808 01:09:05.542500 812 leader_election.cc:302] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters:
I20260808 01:09:05.543376 1053 raft_consensus.cc:2781] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:05.543917 1053 raft_consensus.cc:482] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:05.544358 1053 raft_consensus.cc:3037] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.551291 1053 raft_consensus.cc:504] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.558133 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:05.558899 924 raft_consensus.cc:3037] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.564374 1053 leader_election.cc:288] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 election: Requested vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.566404 924 raft_consensus.cc:2454] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 1.
I20260808 01:09:05.568004 1053 consensus_queue.cc:1035] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:05.568496 812 leader_election.cc:302] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters:
I20260808 01:09:05.568982 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:05.569342 1053 raft_consensus.cc:2781] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:05.569574 1005 raft_consensus.cc:3037] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.569882 1053 raft_consensus.cc:686] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Becoming Leader. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:05.570621 1053 consensus_queue.cc:237] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.578574 1005 raft_consensus.cc:2454] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 1.
I20260808 01:09:05.580775 743 catalog_manager.cc:5495] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a reported cstate change: term changed from 0 to 1, leader changed from <none> to 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65). New cstate: current_term: 1 leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:05.593469 1053 raft_consensus.cc:482] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:05.593967 1053 raft_consensus.cc:504] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.595443 1053 leader_election.cc:288] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.596408 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:05.596721 1004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:05.596969 924 raft_consensus.cc:2454] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 0.
I20260808 01:09:05.597357 1004 raft_consensus.cc:2454] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 0.
I20260808 01:09:05.597923 812 leader_election.cc:302] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters:
I20260808 01:09:05.598591 1053 raft_consensus.cc:2781] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:05.598902 1053 raft_consensus.cc:482] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:05.599184 1053 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.603750 1053 raft_consensus.cc:504] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.605444 1053 leader_election.cc:288] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 election: Requested vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.606647 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:05.606871 1004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:05.607190 924 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.607461 1004 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:05.612138 1004 raft_consensus.cc:2454] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 1.
I20260808 01:09:05.613142 924 raft_consensus.cc:2454] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 1.
I20260808 01:09:05.613209 812 leader_election.cc:302] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:05.614099 1053 raft_consensus.cc:2781] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:05.614562 1053 raft_consensus.cc:686] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Becoming Leader. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:05.615177 1053 consensus_queue.cc:237] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.623289 741 catalog_manager.cc:5495] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a reported cstate change: term changed from 0 to 1, leader changed from <none> to 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65). New cstate: current_term: 1 leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:05.642220 1053 consensus_queue.cc:1035] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:05.654493 1114 consensus_queue.cc:1035] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:05.725739 1114 consensus_queue.cc:1035] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:05.737910 1080 consensus_queue.cc:1035] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:05.786057 1053 consensus_queue.cc:1035] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:05.796814 1053 consensus_queue.cc:1035] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:05.820745 924 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4"
dest_uuid: "3286595befcc479093c91952e8ee51d4"
mode: GRACEFUL
new_leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
from {username='slave'} at 127.0.0.1:43496
I20260808 01:09:05.821476 924 raft_consensus.cc:595] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 1 LEADER]: Received request to transfer leadership to TS 81a73ef3a37a4fcbbba78acd9ca2b95a
I20260808 01:09:05.846660 1004 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490"
dest_uuid: "9ed48023032e41438840c4a91055fd51"
mode: GRACEFUL
new_leader_uuid: "3286595befcc479093c91952e8ee51d4"
from {username='slave'} at 127.0.0.1:55414
I20260808 01:09:05.847365 1004 raft_consensus.cc:595] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 1 LEADER]: Received request to transfer leadership to TS 3286595befcc479093c91952e8ee51d4
I20260808 01:09:05.848408 1102 consensus_queue.cc:1035] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:05.865497 849 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a48d83231e8e4e94a8b0a92791cb861e"
dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
mode: GRACEFUL
new_leader_uuid: "3286595befcc479093c91952e8ee51d4"
from {username='slave'} at 127.0.0.1:42208
I20260808 01:09:05.866057 849 raft_consensus.cc:595] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Received request to transfer leadership to TS 3286595befcc479093c91952e8ee51d4
I20260808 01:09:05.867586 1102 consensus_queue.cc:1035] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:05.872776 849 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d"
dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
mode: GRACEFUL
new_leader_uuid: "3286595befcc479093c91952e8ee51d4"
from {username='slave'} at 127.0.0.1:42208
I20260808 01:09:05.873682 849 raft_consensus.cc:595] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Received request to transfer leadership to TS 3286595befcc479093c91952e8ee51d4
I20260808 01:09:05.881737 849 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01"
dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
mode: GRACEFUL
new_leader_uuid: "3286595befcc479093c91952e8ee51d4"
from {username='slave'} at 127.0.0.1:42208
I20260808 01:09:05.882769 849 raft_consensus.cc:595] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Received request to transfer leadership to TS 3286595befcc479093c91952e8ee51d4
I20260808 01:09:05.885474 1060 raft_consensus.cc:979] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51: : Instructing follower 3286595befcc479093c91952e8ee51d4 to start an election
I20260808 01:09:05.885916 1060 raft_consensus.cc:1067] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 1 LEADER]: Signalling peer 3286595befcc479093c91952e8ee51d4 to start an election
I20260808 01:09:05.887574 924 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490"
dest_uuid: "3286595befcc479093c91952e8ee51d4"
from {username='slave'} at 127.0.0.1:43494
I20260808 01:09:05.887506 1053 consensus_queue.cc:1035] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:05.888253 924 raft_consensus.cc:482] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:05.888707 924 raft_consensus.cc:3037] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:05.892037 849 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "863fef312d0e4f6393c520f611b6af00"
dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
mode: GRACEFUL
new_leader_uuid: "9ed48023032e41438840c4a91055fd51"
from {username='slave'} at 127.0.0.1:42208
I20260808 01:09:05.892684 849 raft_consensus.cc:595] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Received request to transfer leadership to TS 9ed48023032e41438840c4a91055fd51
I20260808 01:09:05.895196 924 raft_consensus.cc:504] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.898226 849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:05.898761 849 raft_consensus.cc:3037] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:05.904829 849 raft_consensus.cc:2454] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 2.
I20260808 01:09:05.905948 890 leader_election.cc:302] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters:
I20260808 01:09:05.906272 924 leader_election.cc:288] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 2 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.906844 1062 raft_consensus.cc:2781] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:09:05.907353 1062 raft_consensus.cc:686] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Becoming Leader. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Running, Role: LEADER
I20260808 01:09:05.907764 1004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:05.908288 1004 raft_consensus.cc:3032] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:09:05.908560 1004 raft_consensus.cc:729] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Running, Role: LEADER
I20260808 01:09:05.908186 1062 consensus_queue.cc:237] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.909178 1004 consensus_queue.cc:260] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.910120 1004 raft_consensus.cc:3037] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:05.912988 1078 consensus_queue.cc:1035] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:05.916946 1004 raft_consensus.cc:2454] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 2.
I20260808 01:09:05.917568 742 catalog_manager.cc:5495] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 reported cstate change: term changed from 1 to 2, leader changed from 9ed48023032e41438840c4a91055fd51 (127.31.212.67) to 3286595befcc479093c91952e8ee51d4 (127.31.212.66). New cstate: current_term: 2 leader_uuid: "3286595befcc479093c91952e8ee51d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:05.923259 1080 raft_consensus.cc:979] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a: : Instructing follower 3286595befcc479093c91952e8ee51d4 to start an election
I20260808 01:09:05.923763 1078 raft_consensus.cc:1067] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Signalling peer 3286595befcc479093c91952e8ee51d4 to start an election
I20260808 01:09:05.924645 1080 consensus_queue.cc:1035] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:05.925457 924 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a48d83231e8e4e94a8b0a92791cb861e"
dest_uuid: "3286595befcc479093c91952e8ee51d4"
from {username='slave'} at 127.0.0.1:43488
I20260808 01:09:05.926026 924 raft_consensus.cc:482] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:05.927464 924 raft_consensus.cc:3037] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:05.931164 1109 consensus_queue.cc:1035] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:05.934007 1080 raft_consensus.cc:979] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a: : Instructing follower 9ed48023032e41438840c4a91055fd51 to start an election
I20260808 01:09:05.934829 1078 raft_consensus.cc:1067] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Signalling peer 9ed48023032e41438840c4a91055fd51 to start an election
I20260808 01:09:05.935154 924 raft_consensus.cc:504] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.941689 848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48d83231e8e4e94a8b0a92791cb861e" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:05.941890 1004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "863fef312d0e4f6393c520f611b6af00"
dest_uuid: "9ed48023032e41438840c4a91055fd51"
from {username='slave'} at 127.0.0.1:55402
I20260808 01:09:05.942353 848 raft_consensus.cc:3032] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:09:05.942471 1004 raft_consensus.cc:482] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:05.942684 848 raft_consensus.cc:729] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:05.942947 1004 raft_consensus.cc:3037] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:05.943574 848 consensus_queue.cc:260] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.944792 848 raft_consensus.cc:3037] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:05.948845 924 leader_election.cc:288] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 2 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:05.950116 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48d83231e8e4e94a8b0a92791cb861e" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:05.950826 1005 raft_consensus.cc:3037] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:05.953249 1078 consensus_queue.cc:1035] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:05.956331 1004 raft_consensus.cc:504] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.960246 1062 raft_consensus.cc:979] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4: : Instructing follower 81a73ef3a37a4fcbbba78acd9ca2b95a to start an election
I20260808 01:09:05.960047 849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863fef312d0e4f6393c520f611b6af00" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:05.960904 849 raft_consensus.cc:3032] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:09:05.961099 1109 raft_consensus.cc:1067] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 1 LEADER]: Signalling peer 81a73ef3a37a4fcbbba78acd9ca2b95a to start an election
I20260808 01:09:05.961370 849 raft_consensus.cc:729] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:05.964270 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863fef312d0e4f6393c520f611b6af00" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:05.969202 846 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4"
dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
from {username='slave'} at 127.0.0.1:42192
I20260808 01:09:05.970135 846 raft_consensus.cc:482] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:05.970661 846 raft_consensus.cc:3037] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:05.979975 1004 leader_election.cc:288] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 2 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:05.982066 1005 raft_consensus.cc:2454] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 2.
I20260808 01:09:05.983702 888 leader_election.cc:302] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:05.984860 1109 raft_consensus.cc:2781] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:09:05.984620 846 raft_consensus.cc:504] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.985525 1109 raft_consensus.cc:686] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Becoming Leader. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Running, Role: LEADER
I20260808 01:09:05.986689 1109 consensus_queue.cc:237] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.990760 849 consensus_queue.cc:260] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:05.992776 849 raft_consensus.cc:3037] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:06.002606 848 raft_consensus.cc:2454] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 2.
I20260808 01:09:06.005817 846 leader_election.cc:288] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 2 election: Requested vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:06.006647 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:06.007359 925 raft_consensus.cc:3032] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:09:06.003813 849 raft_consensus.cc:2454] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 2.
I20260808 01:09:06.008778 968 leader_election.cc:302] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters: 3286595befcc479093c91952e8ee51d4
I20260808 01:09:06.009169 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:06.009756 1005 raft_consensus.cc:3037] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:06.009989 1129 raft_consensus.cc:2781] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:09:06.010485 1129 raft_consensus.cc:686] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [term 2 LEADER]: Becoming Leader. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Running, Role: LEADER
I20260808 01:09:06.011261 1129 consensus_queue.cc:237] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:06.007766 925 raft_consensus.cc:729] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Running, Role: LEADER
I20260808 01:09:06.017340 925 consensus_queue.cc:260] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:06.018358 925 raft_consensus.cc:3037] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:06.018671 743 catalog_manager.cc:5495] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 reported cstate change: term changed from 1 to 2, leader changed from 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65) to 9ed48023032e41438840c4a91055fd51 (127.31.212.67). New cstate: current_term: 2 leader_uuid: "9ed48023032e41438840c4a91055fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:06.023041 1005 raft_consensus.cc:2454] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 2.
I20260808 01:09:06.024504 812 leader_election.cc:302] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:06.025310 1078 raft_consensus.cc:2781] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:09:06.025460 925 raft_consensus.cc:2454] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 2.
I20260808 01:09:06.025856 1078 raft_consensus.cc:686] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 LEADER]: Becoming Leader. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:06.026584 1078 consensus_queue.cc:237] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:06.028682 742 catalog_manager.cc:5495] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 reported cstate change: term changed from 1 to 2, leader changed from 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65) to 3286595befcc479093c91952e8ee51d4 (127.31.212.66). New cstate: current_term: 2 leader_uuid: "3286595befcc479093c91952e8ee51d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:06.034431 743 catalog_manager.cc:5495] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a reported cstate change: term changed from 1 to 2, leader changed from 3286595befcc479093c91952e8ee51d4 (127.31.212.66) to 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65). New cstate: current_term: 2 leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:06.095742 1078 consensus_queue.cc:1035] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:06.106814 1114 consensus_queue.cc:1035] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:06.115284 1053 raft_consensus.cc:979] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a: : Instructing follower 3286595befcc479093c91952e8ee51d4 to start an election
I20260808 01:09:06.115777 1053 raft_consensus.cc:1067] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Signalling peer 3286595befcc479093c91952e8ee51d4 to start an election
I20260808 01:09:06.117393 925 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01"
dest_uuid: "3286595befcc479093c91952e8ee51d4"
from {username='slave'} at 127.0.0.1:43488
I20260808 01:09:06.118273 925 raft_consensus.cc:482] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:06.118682 925 raft_consensus.cc:3037] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:06.123476 925 raft_consensus.cc:504] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:06.125554 925 leader_election.cc:288] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 2 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:06.126757 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:06.126705 849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:06.127434 1005 raft_consensus.cc:3037] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:06.127528 849 raft_consensus.cc:3032] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:09:06.127955 849 raft_consensus.cc:729] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:06.128772 849 consensus_queue.cc:260] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:06.130088 849 raft_consensus.cc:3037] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:06.134541 1005 raft_consensus.cc:2454] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 2.
I20260808 01:09:06.135803 888 leader_election.cc:302] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:06.136629 1062 raft_consensus.cc:2781] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:09:06.136828 849 raft_consensus.cc:2454] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 2.
I20260808 01:09:06.137096 1062 raft_consensus.cc:686] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Becoming Leader. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Running, Role: LEADER
I20260808 01:09:06.138072 1062 consensus_queue.cc:237] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:06.144727 743 catalog_manager.cc:5495] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 reported cstate change: term changed from 1 to 2, leader changed from 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65) to 3286595befcc479093c91952e8ee51d4 (127.31.212.66). New cstate: current_term: 2 leader_uuid: "3286595befcc479093c91952e8ee51d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:06.158116 1053 consensus_queue.cc:1035] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:06.164330 1053 consensus_queue.cc:1035] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:06.165773 1080 raft_consensus.cc:979] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a: : Instructing follower 3286595befcc479093c91952e8ee51d4 to start an election
I20260808 01:09:06.166185 1080 raft_consensus.cc:1067] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Signalling peer 3286595befcc479093c91952e8ee51d4 to start an election
I20260808 01:09:06.169044 924 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d"
dest_uuid: "3286595befcc479093c91952e8ee51d4"
from {username='slave'} at 127.0.0.1:43488
I20260808 01:09:06.169625 924 raft_consensus.cc:482] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:06.170008 924 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:06.177024 924 raft_consensus.cc:504] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:06.180153 924 leader_election.cc:288] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 2 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:06.180048 849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:06.180814 849 raft_consensus.cc:3032] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:09:06.181169 849 raft_consensus.cc:729] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:06.180078 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:06.182164 849 consensus_queue.cc:260] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:06.182698 1005 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:06.183558 849 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:06.189006 1005 raft_consensus.cc:2454] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 2.
I20260808 01:09:06.190079 849 raft_consensus.cc:2454] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 2.
I20260808 01:09:06.190080 888 leader_election.cc:302] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:06.191118 1062 raft_consensus.cc:2781] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:09:06.191601 1062 raft_consensus.cc:686] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Becoming Leader. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Running, Role: LEADER
I20260808 01:09:06.192457 1062 consensus_queue.cc:237] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:06.200785 742 catalog_manager.cc:5495] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 reported cstate change: term changed from 1 to 2, leader changed from 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65) to 3286595befcc479093c91952e8ee51d4 (127.31.212.66). New cstate: current_term: 2 leader_uuid: "3286595befcc479093c91952e8ee51d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:06.396166 1005 raft_consensus.cc:1261] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:06.397104 849 raft_consensus.cc:1261] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Refusing update from remote peer 9ed48023032e41438840c4a91055fd51: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:06.397648 1062 consensus_queue.cc:1035] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:06.398442 1060 consensus_queue.cc:1035] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:06.407529 924 raft_consensus.cc:3037] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:06.407977 846 raft_consensus.cc:1261] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:06.409596 1109 consensus_queue.cc:1035] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:06.418612 924 raft_consensus.cc:1261] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Refusing update from remote peer 9ed48023032e41438840c4a91055fd51: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:06.420344 1060 consensus_queue.cc:1035] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:06.477950 924 raft_consensus.cc:1261] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Refusing update from remote peer 81a73ef3a37a4fcbbba78acd9ca2b95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:06.479204 1053 consensus_queue.cc:1035] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:06.482446 1005 raft_consensus.cc:1261] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:06.483807 1109 consensus_queue.cc:1035] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:06.488300 1004 raft_consensus.cc:1261] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Refusing update from remote peer 81a73ef3a37a4fcbbba78acd9ca2b95a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:06.492899 1078 consensus_queue.cc:1035] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:06.493533 846 raft_consensus.cc:1261] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:06.495075 1109 consensus_queue.cc:1035] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260808 01:09:06.585180 846 raft_consensus.cc:1261] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:06.586651 1109 consensus_queue.cc:1035] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:06.595109 1004 raft_consensus.cc:1261] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:06.596798 1109 consensus_queue.cc:1035] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:06.605430 1004 raft_consensus.cc:1261] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:06.606878 1137 consensus_queue.cc:1035] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:06.614552 846 raft_consensus.cc:1261] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:06.616074 1109 consensus_queue.cc:1035] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:08.929296 32593 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260808 01:09:08.930785 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:08.953087 846 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad"
dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
mode: GRACEFUL
new_leader_uuid: "9ed48023032e41438840c4a91055fd51"
from {username='slave'} at 127.0.0.1:33684
I20260808 01:09:08.954013 846 raft_consensus.cc:595] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Received request to transfer leadership to TS 9ed48023032e41438840c4a91055fd51
I20260808 01:09:08.957125 846 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797"
dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
mode: GRACEFUL
new_leader_uuid: "9ed48023032e41438840c4a91055fd51"
from {username='slave'} at 127.0.0.1:33684
I20260808 01:09:08.957724 846 raft_consensus.cc:595] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Received request to transfer leadership to TS 9ed48023032e41438840c4a91055fd51
I20260808 01:09:08.958891 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 01:09:08.959422 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:09.230509 1153 raft_consensus.cc:979] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a: : Instructing follower 9ed48023032e41438840c4a91055fd51 to start an election
I20260808 01:09:09.231024 1153 raft_consensus.cc:1067] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Signalling peer 9ed48023032e41438840c4a91055fd51 to start an election
I20260808 01:09:09.231483 1078 raft_consensus.cc:979] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a: : Instructing follower 9ed48023032e41438840c4a91055fd51 to start an election
I20260808 01:09:09.232185 1152 raft_consensus.cc:1067] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Signalling peer 9ed48023032e41438840c4a91055fd51 to start an election
I20260808 01:09:09.233112 1005 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad"
dest_uuid: "9ed48023032e41438840c4a91055fd51"
from {username='slave'} at 127.0.0.1:55402
I20260808 01:09:09.233649 1005 raft_consensus.cc:482] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:09.233836 1004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797"
dest_uuid: "9ed48023032e41438840c4a91055fd51"
from {username='slave'} at 127.0.0.1:55402
I20260808 01:09:09.234035 1005 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:09.234407 1004 raft_consensus.cc:482] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:09.234784 1004 raft_consensus.cc:3037] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:09.241027 1005 raft_consensus.cc:504] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:09.241588 1004 raft_consensus.cc:504] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:09.243234 1005 leader_election.cc:288] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 2 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:09.243260 1004 leader_election.cc:288] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 2 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:09.244238 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:09.244801 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:09.245123 846 raft_consensus.cc:3032] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:09:09.245384 924 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:09.245399 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:09.245471 846 raft_consensus.cc:729] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:09.246335 925 raft_consensus.cc:3037] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:09.246454 849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:09.247143 846 consensus_queue.cc:260] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:09.247138 849 raft_consensus.cc:3032] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:09:09.249081 849 raft_consensus.cc:729] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:09.249935 849 consensus_queue.cc:260] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:09.251518 849 raft_consensus.cc:3037] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:09.251554 846 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:09.253784 924 raft_consensus.cc:2454] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 2.
I20260808 01:09:09.255270 966 leader_election.cc:302] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:09.256368 1148 raft_consensus.cc:2781] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:09:09.260418 849 raft_consensus.cc:2454] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 2.
I20260808 01:09:09.261533 846 raft_consensus.cc:2454] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 2.
I20260808 01:09:09.262331 968 leader_election.cc:302] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:09.263098 1148 raft_consensus.cc:686] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 2 LEADER]: Becoming Leader. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Running, Role: LEADER
I20260808 01:09:09.263163 925 raft_consensus.cc:2454] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 2.
I20260808 01:09:09.264447 1148 consensus_queue.cc:237] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:09.265174 1156 raft_consensus.cc:2781] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:09:09.268296 1156 raft_consensus.cc:686] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 2 LEADER]: Becoming Leader. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Running, Role: LEADER
I20260808 01:09:09.269454 1156 consensus_queue.cc:237] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:09.274255 743 catalog_manager.cc:5495] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 reported cstate change: term changed from 1 to 2, leader changed from 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65) to 9ed48023032e41438840c4a91055fd51 (127.31.212.67). New cstate: current_term: 2 leader_uuid: "9ed48023032e41438840c4a91055fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:09.290789 743 catalog_manager.cc:5495] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 reported cstate change: term changed from 1 to 2, leader changed from 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65) to 9ed48023032e41438840c4a91055fd51 (127.31.212.67). New cstate: current_term: 2 leader_uuid: "9ed48023032e41438840c4a91055fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:09.694183 925 raft_consensus.cc:1261] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Refusing update from remote peer 9ed48023032e41438840c4a91055fd51: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:09.695485 1156 consensus_queue.cc:1035] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:09.709456 846 raft_consensus.cc:1261] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Refusing update from remote peer 9ed48023032e41438840c4a91055fd51: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:09.711479 1157 consensus_queue.cc:1035] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:09.847616 925 raft_consensus.cc:1261] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Refusing update from remote peer 9ed48023032e41438840c4a91055fd51: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:09.849225 1157 consensus_queue.cc:1035] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:09.860559 846 raft_consensus.cc:1261] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Refusing update from remote peer 9ed48023032e41438840c4a91055fd51: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:09.863034 1171 consensus_queue.cc:1035] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:09.960433 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:09.966889 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:09.967274 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:10.968317 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:10.974232 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:10.974650 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:11.975589 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:11.981012 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:11.981397 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:12.982236 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:12.988317 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:12.988716 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:13.989629 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:13.994867 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:13.995267 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:14.996157 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:15.002411 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:15.002948 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:16.003957 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:16.009804 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:16.010310 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:17.011153 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:17.017501 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:17.018016 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:18.018980 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:18.024636 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:18.025077 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:19.060324 32593 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260808 01:09:19.073138 1005 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad"
dest_uuid: "9ed48023032e41438840c4a91055fd51"
mode: GRACEFUL
new_leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
from {username='slave'} at 127.0.0.1:55414
I20260808 01:09:19.073930 1005 raft_consensus.cc:595] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 2 LEADER]: Received request to transfer leadership to TS 81a73ef3a37a4fcbbba78acd9ca2b95a
I20260808 01:09:19.079207 1005 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797"
dest_uuid: "9ed48023032e41438840c4a91055fd51"
mode: GRACEFUL
new_leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
from {username='slave'} at 127.0.0.1:55414
I20260808 01:09:19.079836 1005 raft_consensus.cc:595] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 2 LEADER]: Received request to transfer leadership to TS 81a73ef3a37a4fcbbba78acd9ca2b95a
I20260808 01:09:19.084754 925 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490"
dest_uuid: "3286595befcc479093c91952e8ee51d4"
mode: GRACEFUL
new_leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
from {username='slave'} at 127.0.0.1:43496
I20260808 01:09:19.085336 925 raft_consensus.cc:595] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Received request to transfer leadership to TS 81a73ef3a37a4fcbbba78acd9ca2b95a
I20260808 01:09:19.090332 925 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a48d83231e8e4e94a8b0a92791cb861e"
dest_uuid: "3286595befcc479093c91952e8ee51d4"
mode: GRACEFUL
new_leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
from {username='slave'} at 127.0.0.1:43496
I20260808 01:09:19.090767 925 raft_consensus.cc:595] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Received request to transfer leadership to TS 81a73ef3a37a4fcbbba78acd9ca2b95a
I20260808 01:09:19.095108 925 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d"
dest_uuid: "3286595befcc479093c91952e8ee51d4"
mode: GRACEFUL
new_leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
from {username='slave'} at 127.0.0.1:43496
I20260808 01:09:19.095538 925 raft_consensus.cc:595] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Received request to transfer leadership to TS 81a73ef3a37a4fcbbba78acd9ca2b95a
I20260808 01:09:19.160218 1174 raft_consensus.cc:979] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51: : Instructing follower 81a73ef3a37a4fcbbba78acd9ca2b95a to start an election
I20260808 01:09:19.160776 1181 raft_consensus.cc:1067] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 2 LEADER]: Signalling peer 81a73ef3a37a4fcbbba78acd9ca2b95a to start an election
I20260808 01:09:19.162351 846 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad"
dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
from {username='slave'} at 127.0.0.1:42194
I20260808 01:09:19.162827 846 raft_consensus.cc:482] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:19.163174 846 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:19.169222 846 raft_consensus.cc:504] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:19.171072 846 leader_election.cc:288] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 3 election: Requested vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:19.172446 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:19.172672 1004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:19.173314 925 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:19.173794 1004 raft_consensus.cc:3032] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:09:19.174167 1004 raft_consensus.cc:729] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Running, Role: LEADER
I20260808 01:09:19.175120 1004 consensus_queue.cc:260] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:19.176709 1004 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:19.181629 925 raft_consensus.cc:2454] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 3.
I20260808 01:09:19.183007 812 leader_election.cc:302] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters:
I20260808 01:09:19.183998 1177 raft_consensus.cc:2781] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:09:19.184173 1004 raft_consensus.cc:2454] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 3.
I20260808 01:09:19.185196 1177 raft_consensus.cc:686] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Becoming Leader. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:19.186838 1177 consensus_queue.cc:237] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:19.196254 740 catalog_manager.cc:5495] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a reported cstate change: term changed from 2 to 3, leader changed from 9ed48023032e41438840c4a91055fd51 (127.31.212.67) to 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65). New cstate: current_term: 3 leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:19.272771 1174 raft_consensus.cc:979] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51: : Instructing follower 81a73ef3a37a4fcbbba78acd9ca2b95a to start an election
I20260808 01:09:19.273242 1181 raft_consensus.cc:1067] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 2 LEADER]: Signalling peer 81a73ef3a37a4fcbbba78acd9ca2b95a to start an election
I20260808 01:09:19.274664 846 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797"
dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
from {username='slave'} at 127.0.0.1:42194
I20260808 01:09:19.275175 846 raft_consensus.cc:482] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:19.275556 846 raft_consensus.cc:3037] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:19.280931 846 raft_consensus.cc:504] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:19.282177 846 leader_election.cc:288] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 3 election: Requested vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:19.282990 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:19.283254 1004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:19.283571 925 raft_consensus.cc:3037] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:19.283723 1004 raft_consensus.cc:3032] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:09:19.284061 1004 raft_consensus.cc:729] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Running, Role: LEADER
I20260808 01:09:19.284858 1004 consensus_queue.cc:260] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:19.286063 1004 raft_consensus.cc:3037] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:19.290122 925 raft_consensus.cc:2454] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 3.
I20260808 01:09:19.291266 812 leader_election.cc:302] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters:
I20260808 01:09:19.292038 1177 raft_consensus.cc:2781] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:09:19.292371 1004 raft_consensus.cc:2454] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 3.
I20260808 01:09:19.292696 1177 raft_consensus.cc:686] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Becoming Leader. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:19.293691 1177 consensus_queue.cc:237] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:19.296371 1188 raft_consensus.cc:979] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4: : Instructing follower 81a73ef3a37a4fcbbba78acd9ca2b95a to start an election
I20260808 01:09:19.296998 1176 raft_consensus.cc:1067] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Signalling peer 81a73ef3a37a4fcbbba78acd9ca2b95a to start an election
I20260808 01:09:19.298786 846 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490"
dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
from {username='slave'} at 127.0.0.1:42192
I20260808 01:09:19.299321 846 raft_consensus.cc:482] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:19.299655 846 raft_consensus.cc:3037] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:19.302211 740 catalog_manager.cc:5495] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a reported cstate change: term changed from 2 to 3, leader changed from 9ed48023032e41438840c4a91055fd51 (127.31.212.67) to 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65). New cstate: current_term: 3 leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:19.306070 846 raft_consensus.cc:504] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:19.307788 846 leader_election.cc:288] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 3 election: Requested vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:19.308830 1004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:19.309451 1004 raft_consensus.cc:3037] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:19.308861 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:19.311223 925 raft_consensus.cc:3032] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:09:19.311559 925 raft_consensus.cc:729] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Running, Role: LEADER
I20260808 01:09:19.312304 925 consensus_queue.cc:260] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:19.313481 925 raft_consensus.cc:3037] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:19.316026 1004 raft_consensus.cc:2454] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 3.
I20260808 01:09:19.317163 812 leader_election.cc:302] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:19.317906 1177 raft_consensus.cc:2781] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:09:19.318760 1177 raft_consensus.cc:686] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Becoming Leader. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:19.319725 1177 consensus_queue.cc:237] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:19.320654 925 raft_consensus.cc:2454] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 3.
I20260808 01:09:19.322624 1188 raft_consensus.cc:979] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4: : Instructing follower 81a73ef3a37a4fcbbba78acd9ca2b95a to start an election
I20260808 01:09:19.323050 1188 raft_consensus.cc:1067] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Signalling peer 81a73ef3a37a4fcbbba78acd9ca2b95a to start an election
I20260808 01:09:19.324843 846 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d"
dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
from {username='slave'} at 127.0.0.1:42192
I20260808 01:09:19.325552 846 raft_consensus.cc:482] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:19.326014 846 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:19.330605 740 catalog_manager.cc:5495] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a reported cstate change: term changed from 2 to 3, leader changed from 3286595befcc479093c91952e8ee51d4 (127.31.212.66) to 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65). New cstate: current_term: 3 leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:19.333160 846 raft_consensus.cc:504] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:19.334921 846 leader_election.cc:288] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 3 election: Requested vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:19.335832 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:19.335958 1004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:19.336508 925 raft_consensus.cc:3032] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:09:19.336612 1004 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:19.336802 925 raft_consensus.cc:729] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Running, Role: LEADER
I20260808 01:09:19.337551 925 consensus_queue.cc:260] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:19.338703 925 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:19.342988 1004 raft_consensus.cc:2454] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 3.
I20260808 01:09:19.343950 812 leader_election.cc:302] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:19.344640 925 raft_consensus.cc:2454] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 3.
I20260808 01:09:19.344640 1177 raft_consensus.cc:2781] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:09:19.345373 1177 raft_consensus.cc:686] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Becoming Leader. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:19.346132 1177 consensus_queue.cc:237] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:19.353437 740 catalog_manager.cc:5495] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a reported cstate change: term changed from 2 to 3, leader changed from 3286595befcc479093c91952e8ee51d4 (127.31.212.66) to 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65). New cstate: current_term: 3 leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:19.577761 1188 raft_consensus.cc:979] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4: : Instructing follower 81a73ef3a37a4fcbbba78acd9ca2b95a to start an election
I20260808 01:09:19.578274 1176 raft_consensus.cc:1067] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Signalling peer 81a73ef3a37a4fcbbba78acd9ca2b95a to start an election
I20260808 01:09:19.579749 846 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a48d83231e8e4e94a8b0a92791cb861e"
dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
from {username='slave'} at 127.0.0.1:42192
I20260808 01:09:19.580200 846 raft_consensus.cc:482] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:19.580435 846 raft_consensus.cc:3037] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:19.586364 846 raft_consensus.cc:504] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:19.587747 846 leader_election.cc:288] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 3 election: Requested vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:19.588748 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48d83231e8e4e94a8b0a92791cb861e" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:19.588979 1004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48d83231e8e4e94a8b0a92791cb861e" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:19.589344 925 raft_consensus.cc:3032] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:09:19.589437 1004 raft_consensus.cc:3037] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:19.589707 925 raft_consensus.cc:729] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Running, Role: LEADER
I20260808 01:09:19.590440 925 consensus_queue.cc:260] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:19.591599 925 raft_consensus.cc:3037] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:19.596145 1004 raft_consensus.cc:2454] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 3.
I20260808 01:09:19.597249 812 leader_election.cc:302] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:19.597999 1177 raft_consensus.cc:2781] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:09:19.598049 925 raft_consensus.cc:2454] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 3.
I20260808 01:09:19.598528 1177 raft_consensus.cc:686] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Becoming Leader. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:19.599466 1177 consensus_queue.cc:237] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:19.610659 740 catalog_manager.cc:5495] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a reported cstate change: term changed from 2 to 3, leader changed from 3286595befcc479093c91952e8ee51d4 (127.31.212.66) to 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65). New cstate: current_term: 3 leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:19.693248 1004 raft_consensus.cc:1261] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Refusing update from remote peer 81a73ef3a37a4fcbbba78acd9ca2b95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:09:19.694835 1177 consensus_queue.cc:1035] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:19.706382 925 raft_consensus.cc:1261] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Refusing update from remote peer 81a73ef3a37a4fcbbba78acd9ca2b95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:09:19.708101 1177 consensus_queue.cc:1035] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:19.771564 1004 raft_consensus.cc:1261] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Refusing update from remote peer 81a73ef3a37a4fcbbba78acd9ca2b95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:09:19.772775 1177 consensus_queue.cc:1035] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:19.782706 925 raft_consensus.cc:1261] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Refusing update from remote peer 81a73ef3a37a4fcbbba78acd9ca2b95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:09:19.785032 1182 consensus_queue.cc:1035] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:19.824476 1004 raft_consensus.cc:1261] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Refusing update from remote peer 81a73ef3a37a4fcbbba78acd9ca2b95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:09:19.824749 1005 raft_consensus.cc:1261] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Refusing update from remote peer 81a73ef3a37a4fcbbba78acd9ca2b95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:09:19.826119 1200 consensus_queue.cc:1035] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:19.826179 1182 consensus_queue.cc:1035] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:19.837459 925 raft_consensus.cc:1261] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Refusing update from remote peer 81a73ef3a37a4fcbbba78acd9ca2b95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:09:19.839210 1177 consensus_queue.cc:1035] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:19.844359 925 raft_consensus.cc:1261] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Refusing update from remote peer 81a73ef3a37a4fcbbba78acd9ca2b95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:09:19.847246 1205 consensus_queue.cc:1035] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:20.097229 925 raft_consensus.cc:1261] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Refusing update from remote peer 81a73ef3a37a4fcbbba78acd9ca2b95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:09:20.098618 1205 consensus_queue.cc:1035] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:20.108774 1004 raft_consensus.cc:1261] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Refusing update from remote peer 81a73ef3a37a4fcbbba78acd9ca2b95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:09:20.111121 1205 consensus_queue.cc:1035] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260808 01:09:20.517163 1028 debug-util.cc:398] Leaking SignalData structure 0x7b08000adb80 after lost signal to thread 32596
W20260808 01:09:20.518275 1028 debug-util.cc:398] Leaking SignalData structure 0x7b08000f0720 after lost signal to thread 779
W20260808 01:09:20.519237 1028 debug-util.cc:398] Leaking SignalData structure 0x7b08000cc0e0 after lost signal to thread 873
W20260808 01:09:20.520108 1028 debug-util.cc:398] Leaking SignalData structure 0x7b080005c960 after lost signal to thread 951
W20260808 01:09:20.520900 1028 debug-util.cc:398] Leaking SignalData structure 0x7b08001ef2e0 after lost signal to thread 1031
I20260808 01:09:22.238580 32593 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260808 01:09:22.239310 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:22.247388 846 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d"
dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
mode: GRACEFUL
new_leader_uuid: "3286595befcc479093c91952e8ee51d4"
from {username='slave'} at 127.0.0.1:33684
I20260808 01:09:22.247946 846 raft_consensus.cc:595] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Received request to transfer leadership to TS 3286595befcc479093c91952e8ee51d4
I20260808 01:09:22.250625 846 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad"
dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
mode: GRACEFUL
new_leader_uuid: "9ed48023032e41438840c4a91055fd51"
from {username='slave'} at 127.0.0.1:33684
I20260808 01:09:22.251096 846 raft_consensus.cc:595] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Received request to transfer leadership to TS 9ed48023032e41438840c4a91055fd51
I20260808 01:09:22.253444 846 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a6ec8523a0424a6eb9bdd08d1546e64d"
dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
mode: GRACEFUL
new_leader_uuid: "9ed48023032e41438840c4a91055fd51"
from {username='slave'} at 127.0.0.1:33684
I20260808 01:09:22.253953 846 raft_consensus.cc:595] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Received request to transfer leadership to TS 9ed48023032e41438840c4a91055fd51
I20260808 01:09:22.256249 846 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797"
dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
mode: GRACEFUL
new_leader_uuid: "3286595befcc479093c91952e8ee51d4"
from {username='slave'} at 127.0.0.1:33684
I20260808 01:09:22.256829 846 raft_consensus.cc:595] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Received request to transfer leadership to TS 3286595befcc479093c91952e8ee51d4
I20260808 01:09:22.257843 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260808 01:09:22.258589 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:22.337514 1210 raft_consensus.cc:979] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a: : Instructing follower 3286595befcc479093c91952e8ee51d4 to start an election
I20260808 01:09:22.338095 1210 raft_consensus.cc:1067] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Signalling peer 3286595befcc479093c91952e8ee51d4 to start an election
I20260808 01:09:22.339540 1210 raft_consensus.cc:979] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a: : Instructing follower 9ed48023032e41438840c4a91055fd51 to start an election
I20260808 01:09:22.339604 925 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797"
dest_uuid: "3286595befcc479093c91952e8ee51d4"
from {username='slave'} at 127.0.0.1:43488
I20260808 01:09:22.340205 1200 raft_consensus.cc:1067] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Signalling peer 9ed48023032e41438840c4a91055fd51 to start an election
I20260808 01:09:22.340340 925 raft_consensus.cc:482] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:22.340698 925 raft_consensus.cc:3037] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:09:22.341904 1004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad"
dest_uuid: "9ed48023032e41438840c4a91055fd51"
from {username='slave'} at 127.0.0.1:55402
I20260808 01:09:22.342406 1004 raft_consensus.cc:482] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:22.342705 1004 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:09:22.348016 1004 raft_consensus.cc:504] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:22.349578 925 raft_consensus.cc:504] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:22.351218 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:22.352157 925 leader_election.cc:288] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 4 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:22.351763 846 raft_consensus.cc:3032] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:09:22.353801 846 raft_consensus.cc:729] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:22.354077 849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:22.354611 849 raft_consensus.cc:3032] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:09:22.354972 849 raft_consensus.cc:729] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:22.354545 846 consensus_queue.cc:260] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:22.355675 846 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:09:22.356132 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:22.356664 1005 raft_consensus.cc:3037] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:09:22.355708 849 consensus_queue.cc:260] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:22.357887 849 raft_consensus.cc:3037] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:09:22.363416 1005 raft_consensus.cc:2454] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 4.
I20260808 01:09:22.364367 1004 leader_election.cc:288] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 4 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:22.364938 888 leader_election.cc:302] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:22.366122 1176 raft_consensus.cc:2781] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:09:22.369021 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:22.369630 925 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:09:22.373605 849 raft_consensus.cc:2454] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 4.
I20260808 01:09:22.374662 846 raft_consensus.cc:2454] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 4.
I20260808 01:09:22.375950 968 leader_election.cc:302] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:22.376952 1181 raft_consensus.cc:2781] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:09:22.378440 1176 raft_consensus.cc:686] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 4 LEADER]: Becoming Leader. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Running, Role: LEADER
I20260808 01:09:22.379411 1176 consensus_queue.cc:237] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:22.380089 925 raft_consensus.cc:2454] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 4.
I20260808 01:09:22.382359 1181 raft_consensus.cc:686] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 4 LEADER]: Becoming Leader. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Running, Role: LEADER
I20260808 01:09:22.383399 1181 consensus_queue.cc:237] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:22.389504 1200 raft_consensus.cc:979] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a: : Instructing follower 3286595befcc479093c91952e8ee51d4 to start an election
I20260808 01:09:22.390113 1200 raft_consensus.cc:1067] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Signalling peer 3286595befcc479093c91952e8ee51d4 to start an election
I20260808 01:09:22.389936 743 catalog_manager.cc:5495] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 reported cstate change: term changed from 3 to 4, leader changed from 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65) to 3286595befcc479093c91952e8ee51d4 (127.31.212.66). New cstate: current_term: 4 leader_uuid: "3286595befcc479093c91952e8ee51d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:22.392700 925 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d"
dest_uuid: "3286595befcc479093c91952e8ee51d4"
from {username='slave'} at 127.0.0.1:43488
I20260808 01:09:22.393436 925 raft_consensus.cc:482] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:22.393770 925 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:09:22.406286 740 catalog_manager.cc:5495] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 reported cstate change: term changed from 3 to 4, leader changed from 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65) to 9ed48023032e41438840c4a91055fd51 (127.31.212.67). New cstate: current_term: 4 leader_uuid: "9ed48023032e41438840c4a91055fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:22.407338 925 raft_consensus.cc:504] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:22.409222 925 leader_election.cc:288] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 4 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:22.410199 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:22.410233 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:22.411374 1005 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:09:22.410813 846 raft_consensus.cc:3032] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:09:22.415330 846 raft_consensus.cc:729] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:22.416172 846 consensus_queue.cc:260] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:22.417454 846 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:09:22.419575 1005 raft_consensus.cc:2454] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 4.
I20260808 01:09:22.421118 888 leader_election.cc:302] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:22.422241 1176 raft_consensus.cc:2781] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:09:22.422857 1176 raft_consensus.cc:686] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 4 LEADER]: Becoming Leader. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Running, Role: LEADER
I20260808 01:09:22.423977 1176 consensus_queue.cc:237] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:22.424743 846 raft_consensus.cc:2454] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 4.
I20260808 01:09:22.441496 740 catalog_manager.cc:5495] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 reported cstate change: term changed from 3 to 4, leader changed from 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65) to 3286595befcc479093c91952e8ee51d4 (127.31.212.66). New cstate: current_term: 4 leader_uuid: "3286595befcc479093c91952e8ee51d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:22.768217 846 raft_consensus.cc:1261] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:09:22.769726 1176 consensus_queue.cc:1035] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:09:22.778607 1210 raft_consensus.cc:979] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a: : Instructing follower 9ed48023032e41438840c4a91055fd51 to start an election
I20260808 01:09:22.779083 1210 raft_consensus.cc:1067] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Signalling peer 9ed48023032e41438840c4a91055fd51 to start an election
I20260808 01:09:22.780934 1005 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a6ec8523a0424a6eb9bdd08d1546e64d"
dest_uuid: "9ed48023032e41438840c4a91055fd51"
from {username='slave'} at 127.0.0.1:55402
I20260808 01:09:22.781505 1005 raft_consensus.cc:482] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:22.781833 1005 raft_consensus.cc:3037] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:22.789647 1005 raft_consensus.cc:504] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:22.792865 1004 raft_consensus.cc:1261] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:09:22.796085 1212 consensus_queue.cc:1035] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:09:22.798842 1005 leader_election.cc:288] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 2 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:22.800643 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ec8523a0424a6eb9bdd08d1546e64d" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:22.801210 846 raft_consensus.cc:3032] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:09:22.801496 846 raft_consensus.cc:729] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:22.802208 846 consensus_queue.cc:260] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:22.803470 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ec8523a0424a6eb9bdd08d1546e64d" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:22.803805 846 raft_consensus.cc:3037] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:22.804111 925 raft_consensus.cc:3037] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:22.811700 846 raft_consensus.cc:2454] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 2.
I20260808 01:09:22.812907 968 leader_election.cc:302] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:22.813169 925 raft_consensus.cc:2454] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 2.
I20260808 01:09:22.813952 1181 raft_consensus.cc:2781] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:09:22.814427 1181 raft_consensus.cc:686] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [term 2 LEADER]: Becoming Leader. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Running, Role: LEADER
I20260808 01:09:22.815318 1181 consensus_queue.cc:237] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:22.828740 743 catalog_manager.cc:5495] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 reported cstate change: term changed from 1 to 2, leader changed from 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65) to 9ed48023032e41438840c4a91055fd51 (127.31.212.67). New cstate: current_term: 2 leader_uuid: "9ed48023032e41438840c4a91055fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:22.884634 846 raft_consensus.cc:1261] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Refusing update from remote peer 9ed48023032e41438840c4a91055fd51: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:09:22.885847 1181 consensus_queue.cc:1035] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:09:22.894505 925 raft_consensus.cc:1261] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Refusing update from remote peer 9ed48023032e41438840c4a91055fd51: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:09:22.896925 1231 consensus_queue.cc:1035] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:09:22.913518 846 raft_consensus.cc:1261] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:09:22.914886 1212 consensus_queue.cc:1035] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:09:22.923470 1004 raft_consensus.cc:1261] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:09:22.924927 1176 consensus_queue.cc:1035] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:09:23.259544 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:23.264974 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:23.265396 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:23.349256 846 raft_consensus.cc:1261] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Refusing update from remote peer 9ed48023032e41438840c4a91055fd51: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:23.350658 1231 consensus_queue.cc:1035] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:23.359910 925 raft_consensus.cc:1261] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Refusing update from remote peer 9ed48023032e41438840c4a91055fd51: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:23.361439 1213 consensus_queue.cc:1035] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:24.266217 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:24.274120 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:24.274529 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:25.275419 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:25.280843 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:25.281255 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:26.282069 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:26.287824 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:26.288303 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:27.289165 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:27.295058 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:27.295641 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:28.362396 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:28.926667 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:28.927980 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:29.054134 1241 raft_consensus.cc:482] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81a73ef3a37a4fcbbba78acd9ca2b95a)
I20260808 01:09:29.055805 1241 raft_consensus.cc:504] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.061192 1241 leader_election.cc:288] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:29.468216 1244 raft_consensus.cc:482] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81a73ef3a37a4fcbbba78acd9ca2b95a)
I20260808 01:09:29.469494 1244 raft_consensus.cc:504] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.473208 1244 leader_election.cc:288] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:29.480702 1246 raft_consensus.cc:482] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9ed48023032e41438840c4a91055fd51)
I20260808 01:09:29.486158 1243 raft_consensus.cc:482] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9ed48023032e41438840c4a91055fd51)
I20260808 01:09:29.487102 1243 raft_consensus.cc:504] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.489126 1244 raft_consensus.cc:482] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81a73ef3a37a4fcbbba78acd9ca2b95a)
I20260808 01:09:29.490789 1243 leader_election.cc:288] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:29.490648 1244 raft_consensus.cc:504] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } }
I20260808 01:09:29.489640 1246 raft_consensus.cc:504] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.493183 1244 leader_election.cc:288] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:29.494184 1246 leader_election.cc:288] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:29.502575 1243 raft_consensus.cc:482] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 81a73ef3a37a4fcbbba78acd9ca2b95a)
I20260808 01:09:29.503456 1243 raft_consensus.cc:504] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.506055 1243 leader_election.cc:288] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:29.506659 1246 raft_consensus.cc:482] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9ed48023032e41438840c4a91055fd51)
I20260808 01:09:29.507450 1246 raft_consensus.cc:504] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.513707 1246 leader_election.cc:288] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:29.524459 1241 raft_consensus.cc:482] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3286595befcc479093c91952e8ee51d4)
I20260808 01:09:29.525593 1247 raft_consensus.cc:482] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3286595befcc479093c91952e8ee51d4)
I20260808 01:09:29.525519 1241 raft_consensus.cc:504] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.526432 1247 raft_consensus.cc:504] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.530508 1241 leader_election.cc:288] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:29.533066 1248 raft_consensus.cc:482] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 81a73ef3a37a4fcbbba78acd9ca2b95a)
I20260808 01:09:29.533959 1248 raft_consensus.cc:504] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.535075 1241 raft_consensus.cc:482] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3286595befcc479093c91952e8ee51d4)
I20260808 01:09:29.535971 1241 raft_consensus.cc:504] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.537643 1248 leader_election.cc:288] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:29.535068 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:29.540423 925 raft_consensus.cc:2454] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 2.
I20260808 01:09:29.542305 923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:29.546901 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:29.547106 1239 raft_consensus.cc:482] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9ed48023032e41438840c4a91055fd51)
I20260808 01:09:29.547577 1005 raft_consensus.cc:2454] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 2.
I20260808 01:09:29.549937 1241 leader_election.cc:288] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:29.547657 1239 raft_consensus.cc:504] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.552068 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863fef312d0e4f6393c520f611b6af00" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:29.554047 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8232c0f88b94775baedaf083a8bdd0e" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:29.554792 1239 leader_election.cc:288] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:29.555886 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ec8523a0424a6eb9bdd08d1546e64d" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:29.556475 1004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:29.557111 1004 raft_consensus.cc:2454] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 3.
I20260808 01:09:29.559440 1246 raft_consensus.cc:482] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 81a73ef3a37a4fcbbba78acd9ca2b95a)
I20260808 01:09:29.559657 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:29.560396 924 raft_consensus.cc:2454] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 3.
I20260808 01:09:29.560357 1246 raft_consensus.cc:504] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.562083 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:29.563113 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:29.563848 1247 leader_election.cc:288] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:29.564373 888 leader_election.cc:302] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:29.565265 1244 raft_consensus.cc:2781] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:09:29.565729 1244 raft_consensus.cc:482] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 81a73ef3a37a4fcbbba78acd9ca2b95a)
I20260808 01:09:29.566282 1244 raft_consensus.cc:3037] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:29.567227 888 leader_election.cc:302] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:29.568269 1243 raft_consensus.cc:2781] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:09:29.568646 1243 raft_consensus.cc:482] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 81a73ef3a37a4fcbbba78acd9ca2b95a)
I20260808 01:09:29.569042 1243 raft_consensus.cc:3037] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:09:29.572561 1246 leader_election.cc:288] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:29.575068 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:29.577217 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:29.578038 966 leader_election.cc:302] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:29.578140 846 raft_consensus.cc:2454] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 2.
I20260808 01:09:29.580830 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:29.581178 849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:29.581804 847 raft_consensus.cc:2454] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 4.
I20260808 01:09:29.581996 849 raft_consensus.cc:2454] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 4.
I20260808 01:09:29.578612 1244 raft_consensus.cc:504] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.580313 848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:29.582394 1243 raft_consensus.cc:504] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.582793 1251 raft_consensus.cc:482] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3286595befcc479093c91952e8ee51d4)
I20260808 01:09:29.583532 1251 raft_consensus.cc:504] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.585147 1241 raft_consensus.cc:2781] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:09:29.585553 1241 raft_consensus.cc:482] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 81a73ef3a37a4fcbbba78acd9ca2b95a)
I20260808 01:09:29.585954 1241 raft_consensus.cc:3037] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:29.586107 1244 leader_election.cc:288] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 3 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:29.586263 1251 leader_election.cc:288] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:29.591753 923 raft_consensus.cc:1226] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Rejecting Update request from peer 81a73ef3a37a4fcbbba78acd9ca2b95a for earlier term 3. Current term is 4. Ops: []
I20260808 01:09:29.595638 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ec8523a0424a6eb9bdd08d1546e64d" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:29.596017 923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ec8523a0424a6eb9bdd08d1546e64d" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:29.596256 1002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:29.598891 968 leader_election.cc:302] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters: 3286595befcc479093c91952e8ee51d4
I20260808 01:09:29.599715 966 leader_election.cc:302] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:29.600116 1248 raft_consensus.cc:2781] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:09:29.600999 923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:29.600301 968 leader_election.cc:302] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:29.601415 1248 raft_consensus.cc:482] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3286595befcc479093c91952e8ee51d4)
I20260808 01:09:29.601854 1248 raft_consensus.cc:3037] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:29.602216 1253 raft_consensus.cc:2781] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:09:29.602494 968 leader_election.cc:302] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:29.602684 1253 raft_consensus.cc:482] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 81a73ef3a37a4fcbbba78acd9ca2b95a)
I20260808 01:09:29.600582 1241 raft_consensus.cc:504] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.584640 1243 leader_election.cc:288] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 4 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:29.603246 923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:29.603191 1247 raft_consensus.cc:2781] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:09:29.604991 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:29.605284 1001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:29.605463 1239 raft_consensus.cc:482] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9ed48023032e41438840c4a91055fd51)
I20260808 01:09:29.604135 1253 raft_consensus.cc:3037] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:09:29.605700 1002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48d83231e8e4e94a8b0a92791cb861e" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:29.606273 1005 raft_consensus.cc:1226] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Rejecting Update request from peer 81a73ef3a37a4fcbbba78acd9ca2b95a for earlier term 2. Current term is 3. Ops: []
I20260808 01:09:29.607401 1241 leader_election.cc:288] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 3 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:29.608537 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:29.608803 848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:29.612147 923 raft_consensus.cc:1226] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Rejecting Update request from peer 81a73ef3a37a4fcbbba78acd9ca2b95a for earlier term 2. Current term is 3. Ops: []
I20260808 01:09:29.615617 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863fef312d0e4f6393c520f611b6af00" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:29.615463 1248 raft_consensus.cc:504] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.617831 1239 raft_consensus.cc:504] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.618227 849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8232c0f88b94775baedaf083a8bdd0e" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:29.620184 1241 raft_consensus.cc:2781] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:09:29.613180 1247 raft_consensus.cc:482] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3286595befcc479093c91952e8ee51d4)
I20260808 01:09:29.621850 849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:29.620975 848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ec8523a0424a6eb9bdd08d1546e64d" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:29.622239 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:29.622920 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48d83231e8e4e94a8b0a92791cb861e" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:29.629988 846 raft_consensus.cc:2454] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 4.
I20260808 01:09:29.630275 812 leader_election.cc:302] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51
I20260808 01:09:29.623795 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:29.621246 1247 raft_consensus.cc:3037] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:09:29.625468 1239 leader_election.cc:288] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:29.623544 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:29.627138 890 leader_election.cc:302] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3286595befcc479093c91952e8ee51d4; no voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51
I20260808 01:09:29.627547 1252 consensus_queue.cc:1046] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20260808 01:09:29.637620 1252 raft_consensus.cc:3032] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:09:29.637912 890 leader_election.cc:302] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3286595befcc479093c91952e8ee51d4; no voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51
I20260808 01:09:29.638048 1252 raft_consensus.cc:729] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:29.634305 1251 raft_consensus.cc:2726] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:09:29.638083 1248 leader_election.cc:288] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 3 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:29.640391 1243 raft_consensus.cc:2726] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:29.640878 1244 raft_consensus.cc:2726] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:09:29.641497 890 leader_election.cc:302] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3286595befcc479093c91952e8ee51d4; no voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51
I20260808 01:09:29.620602 1241 raft_consensus.cc:482] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3286595befcc479093c91952e8ee51d4)
I20260808 01:09:29.643530 1241 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:09:29.644284 890 leader_election.cc:302] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3286595befcc479093c91952e8ee51d4; no voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51
I20260808 01:09:29.644876 1243 raft_consensus.cc:2726] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:09:29.645336 890 leader_election.cc:302] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters: 9ed48023032e41438840c4a91055fd51
I20260808 01:09:29.645949 1243 raft_consensus.cc:2781] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:09:29.646270 1243 raft_consensus.cc:482] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9ed48023032e41438840c4a91055fd51)
I20260808 01:09:29.646385 890 leader_election.cc:302] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3286595befcc479093c91952e8ee51d4; no voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51
I20260808 01:09:29.646653 1243 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:09:29.647102 1246 raft_consensus.cc:2726] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:09:29.648518 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:29.651552 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:29.642377 923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:29.652971 968 leader_election.cc:302] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ed48023032e41438840c4a91055fd51; no voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a
I20260808 01:09:29.641852 1252 consensus_queue.cc:260] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.643014 1244 raft_consensus.cc:2726] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:09:29.655326 968 leader_election.cc:302] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ed48023032e41438840c4a91055fd51; no voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a
I20260808 01:09:29.660830 1252 raft_consensus.cc:3037] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:09:29.662655 1241 raft_consensus.cc:504] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.664108 1254 raft_consensus.cc:2726] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:09:29.665063 1254 raft_consensus.cc:2726] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:09:29.647544 890 leader_election.cc:302] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3286595befcc479093c91952e8ee51d4; no voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51
I20260808 01:09:29.653954 1253 raft_consensus.cc:504] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.656446 1247 raft_consensus.cc:504] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.667330 812 leader_election.cc:302] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51
I20260808 01:09:29.668146 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:29.668501 1244 raft_consensus.cc:2726] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:09:29.668135 1241 leader_election.cc:288] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 5 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:29.669392 923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:29.674693 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:29.675521 966 leader_election.cc:302] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ed48023032e41438840c4a91055fd51; no voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a
I20260808 01:09:29.672024 1247 leader_election.cc:288] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 5 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:29.672863 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:29.676318 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:29.676913 1256 raft_consensus.cc:2726] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:09:29.673585 1243 raft_consensus.cc:504] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.678289 968 leader_election.cc:302] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ed48023032e41438840c4a91055fd51; no voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a
I20260808 01:09:29.674111 1255 raft_consensus.cc:2726] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:09:29.678212 1257 consensus_queue.cc:1046] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:29.680146 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:29.681209 1249 raft_consensus.cc:3032] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:09:29.681531 1249 raft_consensus.cc:729] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:29.683192 1253 leader_election.cc:288] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 4 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:29.682228 1249 consensus_queue.cc:260] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.683825 1257 consensus_queue.cc:1046] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:29.687112 923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b677dbeb972c4f6f83f791244b162490" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:29.687055 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:29.687929 923 raft_consensus.cc:2379] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9ed48023032e41438840c4a91055fd51 in current term 4: Already voted for candidate 3286595befcc479093c91952e8ee51d4 in this term.
I20260808 01:09:29.683050 1241 raft_consensus.cc:2726] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:09:29.689738 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:29.690464 1243 leader_election.cc:288] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 5 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:29.691047 1249 raft_consensus.cc:3037] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:29.691582 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:29.692203 843 raft_consensus.cc:2454] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 4.
I20260808 01:09:29.692771 888 leader_election.cc:302] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3286595befcc479093c91952e8ee51d4; no voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51
I20260808 01:09:29.693552 1243 raft_consensus.cc:2726] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:09:29.695117 812 leader_election.cc:302] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51
I20260808 01:09:29.695185 968 leader_election.cc:302] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters: 3286595befcc479093c91952e8ee51d4
I20260808 01:09:29.695812 1252 raft_consensus.cc:2726] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:09:29.696517 1241 raft_consensus.cc:2781] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:09:29.696925 1241 raft_consensus.cc:686] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 4 LEADER]: Becoming Leader. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Running, Role: LEADER
I20260808 01:09:29.697894 1241 consensus_queue.cc:237] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.711099 743 catalog_manager.cc:5495] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 reported cstate change: term changed from 3 to 4, leader changed from 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65) to 9ed48023032e41438840c4a91055fd51 (127.31.212.67). New cstate: current_term: 4 leader_uuid: "9ed48023032e41438840c4a91055fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:29.875728 1005 raft_consensus.cc:1226] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Rejecting Update request from peer 3286595befcc479093c91952e8ee51d4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:09:29.876967 1243 consensus_queue.cc:1046] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:09:29.877918 1246 raft_consensus.cc:3032] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:09:29.878276 1246 raft_consensus.cc:729] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Running, Role: LEADER
I20260808 01:09:29.878831 1005 raft_consensus.cc:1226] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Rejecting Update request from peer 3286595befcc479093c91952e8ee51d4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:09:29.879102 1246 consensus_queue.cc:260] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.879935 1243 consensus_queue.cc:1046] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:09:29.880369 1246 raft_consensus.cc:3037] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:09:29.880856 1244 raft_consensus.cc:3032] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:09:29.881165 1244 raft_consensus.cc:729] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Running, Role: LEADER
I20260808 01:09:29.881834 1244 consensus_queue.cc:260] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.882795 1244 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:09:29.920143 923 raft_consensus.cc:1226] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 5 FOLLOWER]: Rejecting Update request from peer 9ed48023032e41438840c4a91055fd51 for earlier term 4. Current term is 5. Ops: []
I20260808 01:09:29.921514 1241 consensus_queue.cc:1046] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:09:29.922441 1248 raft_consensus.cc:3032] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:09:29.922705 1248 raft_consensus.cc:729] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Running, Role: LEADER
I20260808 01:09:29.923316 1248 consensus_queue.cc:260] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.924276 1248 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:09:29.928979 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:29.934378 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:29.934808 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:29.942174 1005 raft_consensus.cc:1226] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Rejecting Update request from peer 3286595befcc479093c91952e8ee51d4 for earlier term 2. Current term is 3. Ops: []
I20260808 01:09:29.943521 1244 consensus_queue.cc:1046] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:29.944402 1246 raft_consensus.cc:3032] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:09:29.944695 1246 raft_consensus.cc:729] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Running, Role: LEADER
I20260808 01:09:29.945407 1246 consensus_queue.cc:260] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:29.946406 1246 raft_consensus.cc:3037] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:30.297540 843 raft_consensus.cc:1261] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Refusing update from remote peer 9ed48023032e41438840c4a91055fd51: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:09:30.298934 1248 consensus_queue.cc:1035] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:09:30.305209 923 raft_consensus.cc:1261] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Refusing update from remote peer 9ed48023032e41438840c4a91055fd51: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:09:30.306605 1248 consensus_queue.cc:1035] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260808 01:09:30.512940 1028 debug-util.cc:398] Leaking SignalData structure 0x7b0800099aa0 after lost signal to thread 32596
W20260808 01:09:30.513798 1028 debug-util.cc:398] Leaking SignalData structure 0x7b08001ef800 after lost signal to thread 779
W20260808 01:09:30.514823 1028 debug-util.cc:398] Leaking SignalData structure 0x7b08000ade00 after lost signal to thread 873
W20260808 01:09:30.515564 1028 debug-util.cc:398] Leaking SignalData structure 0x7b08000ca0a0 after lost signal to thread 951
W20260808 01:09:30.516368 1028 debug-util.cc:398] Leaking SignalData structure 0x7b080009d6a0 after lost signal to thread 1031
I20260808 01:09:30.935657 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:30.941316 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:09:30.941748 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:31.094897 1257 raft_consensus.cc:482] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:31.095672 1257 raft_consensus.cc:504] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.098209 1257 leader_election.cc:288] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:31.099541 923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:31.099995 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:31.100517 923 raft_consensus.cc:2454] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 3.
I20260808 01:09:31.101084 1005 raft_consensus.cc:2454] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 3.
I20260808 01:09:31.102090 812 leader_election.cc:302] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters:
I20260808 01:09:31.103437 1257 raft_consensus.cc:2781] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:09:31.104472 1257 raft_consensus.cc:482] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:31.105490 1257 raft_consensus.cc:3037] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:09:31.111467 1273 raft_consensus.cc:482] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3286595befcc479093c91952e8ee51d4)
I20260808 01:09:31.112666 1273 raft_consensus.cc:504] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.116585 1273 leader_election.cc:288] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:31.117803 923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:31.119098 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:31.120096 1005 raft_consensus.cc:2379] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in current term 5: Already voted for candidate 9ed48023032e41438840c4a91055fd51 in this term.
I20260808 01:09:31.119486 923 raft_consensus.cc:2454] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 5.
I20260808 01:09:31.122159 812 leader_election.cc:302] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters: 9ed48023032e41438840c4a91055fd51
I20260808 01:09:31.123625 1273 raft_consensus.cc:2781] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:09:31.123076 1257 raft_consensus.cc:504] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.124390 1273 raft_consensus.cc:482] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3286595befcc479093c91952e8ee51d4)
I20260808 01:09:31.124917 1273 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:09:31.126345 1257 leader_election.cc:288] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 4 election: Requested vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:31.127810 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:31.128749 1005 raft_consensus.cc:3037] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:09:31.127547 923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d36b6f9b349058a1b30da6ab047f4" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:31.132272 923 raft_consensus.cc:3037] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:09:31.135008 1273 raft_consensus.cc:504] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.138139 1273 leader_election.cc:288] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 5 election: Requested vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:31.139358 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:31.140426 1005 raft_consensus.cc:2454] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 4.
I20260808 01:09:31.143330 812 leader_election.cc:302] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:31.144196 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:31.144800 1273 raft_consensus.cc:2781] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:09:31.145020 923 raft_consensus.cc:2454] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 4.
I20260808 01:09:31.145507 1005 raft_consensus.cc:2379] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in current term 5: Already voted for candidate 9ed48023032e41438840c4a91055fd51 in this term.
I20260808 01:09:31.146140 1273 raft_consensus.cc:686] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 LEADER]: Becoming Leader. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:31.147955 1273 consensus_queue.cc:237] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.151298 924 raft_consensus.cc:2454] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a73ef3a37a4fcbbba78acd9ca2b95a in term 5.
I20260808 01:09:31.152688 812 leader_election.cc:302] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters: 9ed48023032e41438840c4a91055fd51
I20260808 01:09:31.153753 1257 raft_consensus.cc:2781] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:09:31.154752 1257 raft_consensus.cc:686] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 5 LEADER]: Becoming Leader. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:31.156308 1257 consensus_queue.cc:237] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.170116 740 catalog_manager.cc:5495] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:31.171990 740 catalog_manager.cc:5495] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a reported cstate change: term changed from 4 to 5, leader changed from 3286595befcc479093c91952e8ee51d4 (127.31.212.66) to 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65). New cstate: current_term: 5 leader_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:31.227582 1282 raft_consensus.cc:482] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:31.228178 1282 raft_consensus.cc:504] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.230371 1272 raft_consensus.cc:482] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:31.231273 1282 leader_election.cc:288] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:31.231070 1272 raft_consensus.cc:504] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.231987 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:31.232303 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:31.233007 1005 raft_consensus.cc:2454] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 5.
I20260808 01:09:31.232972 843 raft_consensus.cc:2454] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 4.
I20260808 01:09:31.234571 1272 leader_election.cc:288] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:31.235224 890 leader_election.cc:302] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters:
I20260808 01:09:31.235886 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:31.236128 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:31.236558 1282 raft_consensus.cc:2781] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:09:31.237056 843 raft_consensus.cc:2454] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 4.
I20260808 01:09:31.237143 924 raft_consensus.cc:2454] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 5.
I20260808 01:09:31.237293 1282 raft_consensus.cc:482] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:31.238202 1282 raft_consensus.cc:3037] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:09:31.238636 968 leader_election.cc:302] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:31.239960 1272 raft_consensus.cc:2781] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:09:31.240504 1272 raft_consensus.cc:482] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:31.240949 1272 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:09:31.247432 1282 raft_consensus.cc:504] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.248744 1272 raft_consensus.cc:504] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.250301 1282 leader_election.cc:288] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 6 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:31.251757 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:31.251948 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38325e90b3d4db983aae4c25685e797" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:31.252554 1272 leader_election.cc:288] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 6 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:31.252995 843 raft_consensus.cc:3037] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Advancing to term 6
I20260808 01:09:31.253149 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:31.253151 1005 raft_consensus.cc:3037] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:09:31.254258 844 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Advancing to term 6
I20260808 01:09:31.257055 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:31.264359 924 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:09:31.265064 1005 raft_consensus.cc:2454] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 6.
I20260808 01:09:31.265120 843 raft_consensus.cc:2454] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 6.
I20260808 01:09:31.265621 844 raft_consensus.cc:2454] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 6.
I20260808 01:09:31.266824 888 leader_election.cc:302] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:31.268059 1282 raft_consensus.cc:2781] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:09:31.268694 968 leader_election.cc:302] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a73ef3a37a4fcbbba78acd9ca2b95a, 9ed48023032e41438840c4a91055fd51; no voters:
I20260808 01:09:31.270328 1282 raft_consensus.cc:686] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 6 LEADER]: Becoming Leader. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Running, Role: LEADER
I20260808 01:09:31.270382 1272 raft_consensus.cc:2781] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:09:31.271731 1282 consensus_queue.cc:237] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.272377 1272 raft_consensus.cc:686] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 6 LEADER]: Becoming Leader. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Running, Role: LEADER
I20260808 01:09:31.273994 1272 consensus_queue.cc:237] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.276134 924 raft_consensus.cc:2454] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 6.
I20260808 01:09:31.288515 740 catalog_manager.cc:5495] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "3286595befcc479093c91952e8ee51d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:31.289374 743 catalog_manager.cc:5495] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "9ed48023032e41438840c4a91055fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:31.330176 1282 raft_consensus.cc:482] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:31.330955 1282 raft_consensus.cc:504] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.333932 1282 leader_election.cc:288] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:31.335312 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:31.335602 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:31.336616 1005 raft_consensus.cc:2454] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 3.
I20260808 01:09:31.337916 888 leader_election.cc:302] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51; no voters: 81a73ef3a37a4fcbbba78acd9ca2b95a
I20260808 01:09:31.338831 1282 raft_consensus.cc:2781] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:09:31.339350 1282 raft_consensus.cc:482] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:31.339785 1282 raft_consensus.cc:3037] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:09:31.348595 1282 raft_consensus.cc:504] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.351203 1282 leader_election.cc:288] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 4 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:31.352509 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:31.352658 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:31.353636 1005 raft_consensus.cc:3037] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:09:31.361311 1005 raft_consensus.cc:2454] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 4.
I20260808 01:09:31.362986 888 leader_election.cc:302] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51; no voters: 81a73ef3a37a4fcbbba78acd9ca2b95a
I20260808 01:09:31.364233 1282 raft_consensus.cc:2781] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:09:31.364980 1282 raft_consensus.cc:686] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 4 LEADER]: Becoming Leader. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Running, Role: LEADER
I20260808 01:09:31.366106 1282 consensus_queue.cc:237] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.377202 1273 raft_consensus.cc:482] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3286595befcc479093c91952e8ee51d4)
I20260808 01:09:31.376892 743 catalog_manager.cc:5495] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "3286595befcc479093c91952e8ee51d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:31.378131 1273 raft_consensus.cc:504] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.381861 1273 leader_election.cc:288] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:31.383002 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:31.383443 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52df16f25154c81adfb9d5b50d8bf01" candidate_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9ed48023032e41438840c4a91055fd51" is_pre_election: true
I20260808 01:09:31.384691 1005 raft_consensus.cc:2354] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 81a73ef3a37a4fcbbba78acd9ca2b95a for earlier term 3. Current term is 4.
I20260808 01:09:31.384505 1272 raft_consensus.cc:482] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:31.385468 1272 raft_consensus.cc:504] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.386777 812 leader_election.cc:398] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 3 pre-election: Vote denied by peer 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931) with higher term. Message: Invalid argument: T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 81a73ef3a37a4fcbbba78acd9ca2b95a for earlier term 3. Current term is 4.
I20260808 01:09:31.387163 812 leader_election.cc:401] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:09:31.388003 1273 raft_consensus.cc:3037] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Advancing to term 4
I20260808 01:09:31.390089 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" is_pre_election: true
I20260808 01:09:31.390815 1272 leader_election.cc:288] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:31.392561 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4" is_pre_election: true
I20260808 01:09:31.393596 924 raft_consensus.cc:2454] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 5.
I20260808 01:09:31.395380 966 leader_election.cc:302] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51; no voters: 81a73ef3a37a4fcbbba78acd9ca2b95a
I20260808 01:09:31.396677 1272 raft_consensus.cc:2781] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:09:31.397162 1272 raft_consensus.cc:482] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:31.397683 1272 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:09:31.400068 1273 raft_consensus.cc:2726] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931) with higher term. Message: Invalid argument: T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 81a73ef3a37a4fcbbba78acd9ca2b95a for earlier term 3. Current term is 4.
I20260808 01:09:31.407611 1272 raft_consensus.cc:504] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.411129 1272 leader_election.cc:288] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 6 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485)
I20260808 01:09:31.412920 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3286595befcc479093c91952e8ee51d4"
I20260808 01:09:31.412742 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c615b0358fb4ffd8dd027ce0d401f2d" candidate_uuid: "9ed48023032e41438840c4a91055fd51" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:31.413900 924 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:09:31.423167 924 raft_consensus.cc:2454] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed48023032e41438840c4a91055fd51 in term 6.
I20260808 01:09:31.424863 966 leader_election.cc:302] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 9ed48023032e41438840c4a91055fd51; no voters: 81a73ef3a37a4fcbbba78acd9ca2b95a
I20260808 01:09:31.425971 1272 raft_consensus.cc:2781] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:09:31.426895 1272 raft_consensus.cc:686] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 6 LEADER]: Becoming Leader. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Running, Role: LEADER
I20260808 01:09:31.428268 1272 consensus_queue.cc:237] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.441788 743 catalog_manager.cc:5495] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 reported cstate change: term changed from 5 to 6, leader changed from 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65) to 9ed48023032e41438840c4a91055fd51 (127.31.212.67). New cstate: current_term: 6 leader_uuid: "9ed48023032e41438840c4a91055fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:31.538427 1005 raft_consensus.cc:1226] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 6 LEADER]: Rejecting Update request from peer 81a73ef3a37a4fcbbba78acd9ca2b95a for earlier term 5. Current term is 6. Ops: []
I20260808 01:09:31.540282 1273 consensus_queue.cc:1046] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:09:31.541301 1273 consensus_queue.cc:1225] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Peer 9ed48023032e41438840c4a91055fd51 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260808 01:09:31.541973 1257 raft_consensus.cc:3032] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:09:31.542649 1257 raft_consensus.cc:729] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 81a73ef3a37a4fcbbba78acd9ca2b95a, State: Running, Role: LEADER
I20260808 01:09:31.544003 1257 consensus_queue.cc:260] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:31.546015 1257 raft_consensus.cc:3037] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:09:31.584215 1005 raft_consensus.cc:1261] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Refusing update from remote peer 81a73ef3a37a4fcbbba78acd9ca2b95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:09:31.586310 1273 consensus_queue.cc:1035] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:31.599109 924 raft_consensus.cc:1261] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Refusing update from remote peer 81a73ef3a37a4fcbbba78acd9ca2b95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:09:31.601604 1257 consensus_queue.cc:1035] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:31.677563 844 raft_consensus.cc:1261] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 6 FOLLOWER]: Refusing update from remote peer 9ed48023032e41438840c4a91055fd51: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:09:31.678252 843 raft_consensus.cc:1261] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 6 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:09:31.680778 1272 consensus_queue.cc:1035] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:09:31.681754 1282 consensus_queue.cc:1035] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:09:31.701581 924 raft_consensus.cc:1261] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 6 FOLLOWER]: Refusing update from remote peer 9ed48023032e41438840c4a91055fd51: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:09:31.704058 1272 consensus_queue.cc:1035] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:09:31.705791 1005 raft_consensus.cc:1261] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 6 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:09:31.709306 1282 consensus_queue.cc:1035] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:09:31.754217 843 raft_consensus.cc:1261] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:09:31.756420 1300 consensus_queue.cc:1035] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:31.773125 1005 raft_consensus.cc:1261] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:09:31.776494 1297 consensus_queue.cc:1035] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:31.884656 924 raft_consensus.cc:1261] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 6 FOLLOWER]: Refusing update from remote peer 9ed48023032e41438840c4a91055fd51: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:09:31.885820 1299 consensus_queue.cc:1035] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:09:31.896780 843 raft_consensus.cc:1261] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 6 FOLLOWER]: Refusing update from remote peer 9ed48023032e41438840c4a91055fd51: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260808 01:09:31.897243 843 pending_rounds.cc:76] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a: Aborting all ops after (but not including) 4
I20260808 01:09:31.897535 843 pending_rounds.cc:97] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260808 01:09:31.897878 843 raft_consensus.cc:2866] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:09:31.899320 1299 consensus_queue.cc:1035] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:09:31.942641 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:09:31.964185 1005 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad"
dest_uuid: "9ed48023032e41438840c4a91055fd51"
mode: GRACEFUL
new_leader_uuid: "3286595befcc479093c91952e8ee51d4"
from {username='slave'} at 127.0.0.1:42106
I20260808 01:09:31.964784 1005 raft_consensus.cc:595] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 6 LEADER]: Received request to transfer leadership to TS 3286595befcc479093c91952e8ee51d4
I20260808 01:09:31.965929 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260808 01:09:31.966382 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:09:32.151330 1296 raft_consensus.cc:979] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51: : Instructing follower 3286595befcc479093c91952e8ee51d4 to start an election
I20260808 01:09:32.151908 1299 raft_consensus.cc:1067] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 6 LEADER]: Signalling peer 3286595befcc479093c91952e8ee51d4 to start an election
I20260808 01:09:32.153445 924 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad"
dest_uuid: "3286595befcc479093c91952e8ee51d4"
from {username='slave'} at 127.0.0.1:43494
I20260808 01:09:32.153921 924 raft_consensus.cc:482] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:09:32.154264 924 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:09:32.159915 924 raft_consensus.cc:504] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:32.161574 924 leader_election.cc:288] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 7 election: Requested vote from peers 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159), 9ed48023032e41438840c4a91055fd51 (127.31.212.67:32931)
I20260808 01:09:32.163341 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a"
I20260808 01:09:32.163977 843 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:09:32.163717 1005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9158c2015bb449e99d8c2f8bc251c0ad" candidate_uuid: "3286595befcc479093c91952e8ee51d4" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "9ed48023032e41438840c4a91055fd51"
I20260808 01:09:32.164428 1005 raft_consensus.cc:3032] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:09:32.164752 1005 raft_consensus.cc:729] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9ed48023032e41438840c4a91055fd51, State: Running, Role: LEADER
I20260808 01:09:32.165508 1005 consensus_queue.cc:260] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:32.166661 1005 raft_consensus.cc:3037] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:09:32.171295 843 raft_consensus.cc:2454] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 7.
I20260808 01:09:32.171929 1005 raft_consensus.cc:2454] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3286595befcc479093c91952e8ee51d4 in term 7.
I20260808 01:09:32.172331 890 leader_election.cc:302] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3286595befcc479093c91952e8ee51d4, 81a73ef3a37a4fcbbba78acd9ca2b95a; no voters:
I20260808 01:09:32.173048 1300 raft_consensus.cc:2781] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:09:32.173470 1300 raft_consensus.cc:686] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 7 LEADER]: Becoming Leader. State: Replica: 3286595befcc479093c91952e8ee51d4, State: Running, Role: LEADER
I20260808 01:09:32.174165 1300 consensus_queue.cc:237] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } }
I20260808 01:09:32.180611 743 catalog_manager.cc:5495] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 reported cstate change: term changed from 6 to 7, leader changed from 9ed48023032e41438840c4a91055fd51 (127.31.212.67) to 3286595befcc479093c91952e8ee51d4 (127.31.212.66). New cstate: current_term: 7 leader_uuid: "3286595befcc479093c91952e8ee51d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3286595befcc479093c91952e8ee51d4" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 38485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:32.560686 843 raft_consensus.cc:1261] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 7 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260808 01:09:32.561995 1300 consensus_queue.cc:1035] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a73ef3a37a4fcbbba78acd9ca2b95a" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:09:32.573241 1005 raft_consensus.cc:1261] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 7 FOLLOWER]: Refusing update from remote peer 3286595befcc479093c91952e8ee51d4: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260808 01:09:32.574565 1311 consensus_queue.cc:1035] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed48023032e41438840c4a91055fd51" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:09:32.998044 32593 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260808 01:09:32.998623 32593 tablet_server.cc:178] TabletServer@127.31.212.65:0 shutting down...
I20260808 01:09:33.024842 32593 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260808 01:09:33.025581 32593 tablet_replica.cc:330] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a: stopping tablet replica
I20260808 01:09:33.026162 32593 raft_consensus.cc:2229] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.026594 32593 raft_consensus.cc:2258] T c52df16f25154c81adfb9d5b50d8bf01 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.028453 32593 tablet_replica.cc:330] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a: stopping tablet replica
I20260808 01:09:33.028910 32593 raft_consensus.cc:2229] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.029383 32593 raft_consensus.cc:2258] T a6ec8523a0424a6eb9bdd08d1546e64d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.031140 32593 tablet_replica.cc:330] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a: stopping tablet replica
I20260808 01:09:33.031719 32593 raft_consensus.cc:2229] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.032171 32593 raft_consensus.cc:2258] T 863fef312d0e4f6393c520f611b6af00 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.034068 32593 tablet_replica.cc:330] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a: stopping tablet replica
I20260808 01:09:33.034559 32593 raft_consensus.cc:2229] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 LEADER]: Raft consensus shutting down.
I20260808 01:09:33.035408 32593 raft_consensus.cc:2258] T a48d83231e8e4e94a8b0a92791cb861e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.037178 32593 tablet_replica.cc:330] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a: stopping tablet replica
I20260808 01:09:33.037890 32593 raft_consensus.cc:2229] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 6 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.038318 32593 raft_consensus.cc:2258] T b38325e90b3d4db983aae4c25685e797 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 6 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.040190 32593 tablet_replica.cc:330] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a: stopping tablet replica
I20260808 01:09:33.040686 32593 raft_consensus.cc:2229] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 6 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.041101 32593 raft_consensus.cc:2258] T 3c615b0358fb4ffd8dd027ce0d401f2d P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 6 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.042619 32593 tablet_replica.cc:330] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a: stopping tablet replica
I20260808 01:09:33.043102 32593 raft_consensus.cc:2229] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 7 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.043514 32593 raft_consensus.cc:2258] T 9158c2015bb449e99d8c2f8bc251c0ad P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 7 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.045639 32593 tablet_replica.cc:330] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a: stopping tablet replica
I20260808 01:09:33.046430 32593 raft_consensus.cc:2229] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 LEADER]: Raft consensus shutting down.
I20260808 01:09:33.047276 32593 raft_consensus.cc:2258] T 019d36b6f9b349058a1b30da6ab047f4 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.048904 32593 tablet_replica.cc:330] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a: stopping tablet replica
I20260808 01:09:33.049335 32593 raft_consensus.cc:2229] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.049706 32593 raft_consensus.cc:2258] T b677dbeb972c4f6f83f791244b162490 P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.051158 32593 tablet_replica.cc:330] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a: stopping tablet replica
I20260808 01:09:33.051566 32593 raft_consensus.cc:2229] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 LEADER]: Raft consensus shutting down.
I20260808 01:09:33.052306 32593 raft_consensus.cc:2258] T a8232c0f88b94775baedaf083a8bdd0e P 81a73ef3a37a4fcbbba78acd9ca2b95a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.082824 32593 tablet_server.cc:194] TabletServer@127.31.212.65:0 shutdown complete.
I20260808 01:09:33.101752 32593 tablet_server.cc:178] TabletServer@127.31.212.66:0 shutting down...
W20260808 01:09:33.118251 890 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.212.65:40159: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260808 01:09:33.122558 890 consensus_peers.cc:487] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 -> Peer 81a73ef3a37a4fcbbba78acd9ca2b95a (127.31.212.65:40159): Couldn't send request to peer 81a73ef3a37a4fcbbba78acd9ca2b95a. Status: Network error: Client connection negotiation failed: client connection to 127.31.212.65:40159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:33.128692 32593 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260808 01:09:33.129393 32593 tablet_replica.cc:330] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4: stopping tablet replica
I20260808 01:09:33.130146 32593 raft_consensus.cc:2229] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 4 LEADER]: Raft consensus shutting down.
I20260808 01:09:33.130867 32593 raft_consensus.cc:2258] T c52df16f25154c81adfb9d5b50d8bf01 P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.132644 32593 tablet_replica.cc:330] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4: stopping tablet replica
I20260808 01:09:33.133211 32593 raft_consensus.cc:2229] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.133669 32593 raft_consensus.cc:2258] T b677dbeb972c4f6f83f791244b162490 P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.135316 32593 tablet_replica.cc:330] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4: stopping tablet replica
I20260808 01:09:33.135792 32593 raft_consensus.cc:2229] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.136252 32593 raft_consensus.cc:2258] T a8232c0f88b94775baedaf083a8bdd0e P 3286595befcc479093c91952e8ee51d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.137800 32593 tablet_replica.cc:330] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4: stopping tablet replica
I20260808 01:09:33.138338 32593 raft_consensus.cc:2229] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.138722 32593 raft_consensus.cc:2258] T 3c615b0358fb4ffd8dd027ce0d401f2d P 3286595befcc479093c91952e8ee51d4 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.140431 32593 tablet_replica.cc:330] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4: stopping tablet replica
I20260808 01:09:33.140888 32593 raft_consensus.cc:2229] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 7 LEADER]: Raft consensus shutting down.
I20260808 01:09:33.141587 32593 raft_consensus.cc:2258] T 9158c2015bb449e99d8c2f8bc251c0ad P 3286595befcc479093c91952e8ee51d4 [term 7 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.144268 32593 tablet_replica.cc:330] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4: stopping tablet replica
I20260808 01:09:33.144788 32593 raft_consensus.cc:2229] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.145190 32593 raft_consensus.cc:2258] T 019d36b6f9b349058a1b30da6ab047f4 P 3286595befcc479093c91952e8ee51d4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.146639 32593 tablet_replica.cc:330] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4: stopping tablet replica
I20260808 01:09:33.147110 32593 raft_consensus.cc:2229] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.147507 32593 raft_consensus.cc:2258] T 863fef312d0e4f6393c520f611b6af00 P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.149118 32593 tablet_replica.cc:330] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4: stopping tablet replica
I20260808 01:09:33.149564 32593 raft_consensus.cc:2229] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.150002 32593 raft_consensus.cc:2258] T a48d83231e8e4e94a8b0a92791cb861e P 3286595befcc479093c91952e8ee51d4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.151691 32593 tablet_replica.cc:330] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4: stopping tablet replica
I20260808 01:09:33.152225 32593 raft_consensus.cc:2229] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 6 LEADER]: Raft consensus shutting down.
I20260808 01:09:33.152984 32593 raft_consensus.cc:2258] T b38325e90b3d4db983aae4c25685e797 P 3286595befcc479093c91952e8ee51d4 [term 6 FOLLOWER]: Raft consensus is shut down!
W20260808 01:09:33.153398 966 consensus_peers.cc:487] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 -> Peer 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485): Couldn't send request to peer 3286595befcc479093c91952e8ee51d4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:33.154660 32593 tablet_replica.cc:330] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4: stopping tablet replica
I20260808 01:09:33.155167 32593 raft_consensus.cc:2229] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.155606 32593 raft_consensus.cc:2258] T a6ec8523a0424a6eb9bdd08d1546e64d P 3286595befcc479093c91952e8ee51d4 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260808 01:09:33.162089 966 consensus_peers.cc:487] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 -> Peer 3286595befcc479093c91952e8ee51d4 (127.31.212.66:38485): Couldn't send request to peer 3286595befcc479093c91952e8ee51d4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:33.187040 32593 tablet_server.cc:194] TabletServer@127.31.212.66:0 shutdown complete.
I20260808 01:09:33.206118 32593 tablet_server.cc:178] TabletServer@127.31.212.67:0 shutting down...
I20260808 01:09:33.231669 32593 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260808 01:09:33.232465 32593 tablet_replica.cc:330] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51: stopping tablet replica
I20260808 01:09:33.233100 32593 raft_consensus.cc:2229] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.233553 32593 raft_consensus.cc:2258] T c52df16f25154c81adfb9d5b50d8bf01 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.235251 32593 tablet_replica.cc:330] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51: stopping tablet replica
I20260808 01:09:33.235904 32593 raft_consensus.cc:2229] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 6 LEADER]: Raft consensus shutting down.
I20260808 01:09:33.236662 32593 raft_consensus.cc:2258] T 3c615b0358fb4ffd8dd027ce0d401f2d P 9ed48023032e41438840c4a91055fd51 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.238298 32593 tablet_replica.cc:330] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51: stopping tablet replica
I20260808 01:09:33.238827 32593 raft_consensus.cc:2229] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 7 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.239253 32593 raft_consensus.cc:2258] T 9158c2015bb449e99d8c2f8bc251c0ad P 9ed48023032e41438840c4a91055fd51 [term 7 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.241834 32593 tablet_replica.cc:330] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51: stopping tablet replica
I20260808 01:09:33.242406 32593 raft_consensus.cc:2229] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.242825 32593 raft_consensus.cc:2258] T 019d36b6f9b349058a1b30da6ab047f4 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.244405 32593 tablet_replica.cc:330] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51: stopping tablet replica
I20260808 01:09:33.244879 32593 raft_consensus.cc:2229] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [term 2 LEADER]: Raft consensus shutting down.
I20260808 01:09:33.245643 32593 raft_consensus.cc:2258] T a6ec8523a0424a6eb9bdd08d1546e64d P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.247179 32593 tablet_replica.cc:330] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51: stopping tablet replica
I20260808 01:09:33.247658 32593 raft_consensus.cc:2229] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [term 2 LEADER]: Raft consensus shutting down.
I20260808 01:09:33.248426 32593 raft_consensus.cc:2258] T 863fef312d0e4f6393c520f611b6af00 P 9ed48023032e41438840c4a91055fd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.249986 32593 tablet_replica.cc:330] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51: stopping tablet replica
I20260808 01:09:33.250437 32593 raft_consensus.cc:2229] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.250847 32593 raft_consensus.cc:2258] T a48d83231e8e4e94a8b0a92791cb861e P 9ed48023032e41438840c4a91055fd51 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.252595 32593 tablet_replica.cc:330] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51: stopping tablet replica
I20260808 01:09:33.253062 32593 raft_consensus.cc:2229] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.253463 32593 raft_consensus.cc:2258] T b38325e90b3d4db983aae4c25685e797 P 9ed48023032e41438840c4a91055fd51 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.255052 32593 tablet_replica.cc:330] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51: stopping tablet replica
I20260808 01:09:33.255550 32593 raft_consensus.cc:2229] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 4 LEADER]: Raft consensus shutting down.
I20260808 01:09:33.256328 32593 raft_consensus.cc:2258] T b677dbeb972c4f6f83f791244b162490 P 9ed48023032e41438840c4a91055fd51 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.257877 32593 tablet_replica.cc:330] T a8232c0f88b94775baedaf083a8bdd0e P 9ed48023032e41438840c4a91055fd51: stopping tablet replica
I20260808 01:09:33.258298 32593 raft_consensus.cc:2229] T a8232c0f88b94775baedaf083a8bdd0e P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:33.258692 32593 raft_consensus.cc:2258] T a8232c0f88b94775baedaf083a8bdd0e P 9ed48023032e41438840c4a91055fd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.307101 32593 tablet_server.cc:194] TabletServer@127.31.212.67:0 shutdown complete.
I20260808 01:09:33.327117 32593 master.cc:537] Master@127.31.212.126:36597 shutting down...
I20260808 01:09:33.351287 32593 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [term 1 LEADER]: Raft consensus shutting down.
I20260808 01:09:33.351824 32593 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:33.352173 32593 tablet_replica.cc:330] T 00000000000000000000000000000000 P 375c7e95bb3b47249db008b5db57a83a: stopping tablet replica
I20260808 01:09:33.371779 32593 master.cc:552] Master@127.31.212.126:36597 shutdown complete.
3286595befcc479093c91952e8ee51d4 4
81a73ef3a37a4fcbbba78acd9ca2b95a 5
9ed48023032e41438840c4a91055fd51 1
3286595befcc479093c91952e8ee51d4 4
81a73ef3a37a4fcbbba78acd9ca2b95a 3
9ed48023032e41438840c4a91055fd51 3
3286595befcc479093c91952e8ee51d4 1
81a73ef3a37a4fcbbba78acd9ca2b95a 8
9ed48023032e41438840c4a91055fd51 1
3286595befcc479093c91952e8ee51d4 3
81a73ef3a37a4fcbbba78acd9ca2b95a 3
9ed48023032e41438840c4a91055fd51 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29522 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260808 01:09:33.417029 32593 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.31.212.126:33415
I20260808 01:09:33.418241 32593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 01:09:33.425233 1318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:09:33.425457 1317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:09:33.432590 1320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:09:33.435314 32593 server_base.cc:1034] running on GCE node
I20260808 01:09:33.436560 32593 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:09:33.436789 32593 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:09:33.436944 32593 hybrid_clock.cc:648] HybridClock initialized: now 1786151373436926 us; error 0 us; skew 500 ppm
I20260808 01:09:33.437505 32593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:09:33.440196 32593 webserver.cc:460] Webserver started at http://127.31.212.126:42043/ using document root <none> and password file <none>
I20260808 01:09:33.440677 32593 fs_manager.cc:359] Metadata directory not provided
I20260808 01:09:33.440871 32593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:09:33.441174 32593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:09:33.442355 32593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/master-0-root/instance:
uuid: "5a043bcc76cc4126b15e712346791f23"
format_stamp: "Formatted at 2026-08-08 01:09:33 on dist-test-slave-mvvj"
I20260808 01:09:33.447141 32593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 01:09:33.450662 1325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:33.451576 32593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 01:09:33.451825 32593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/master-0-root
uuid: "5a043bcc76cc4126b15e712346791f23"
format_stamp: "Formatted at 2026-08-08 01:09:33 on dist-test-slave-mvvj"
I20260808 01:09:33.452142 32593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:09:33.466403 32593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 01:09:33.467659 32593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:09:33.506672 32593 rpc_server.cc:307] RPC server started. Bound to: 127.31.212.126:33415
I20260808 01:09:33.506791 1376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.212.126:33415 every 8 connection(s)
I20260808 01:09:33.510807 1377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:33.520376 1377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23: Bootstrap starting.
I20260808 01:09:33.524722 1377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:33.528990 1377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23: No bootstrap required, opened a new log
I20260808 01:09:33.531075 1377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a043bcc76cc4126b15e712346791f23" member_type: VOTER }
I20260808 01:09:33.531471 1377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:33.531659 1377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a043bcc76cc4126b15e712346791f23, State: Initialized, Role: FOLLOWER
I20260808 01:09:33.532325 1377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a043bcc76cc4126b15e712346791f23" member_type: VOTER }
I20260808 01:09:33.532752 1377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 01:09:33.532940 1377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 01:09:33.533151 1377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:33.537144 1377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a043bcc76cc4126b15e712346791f23" member_type: VOTER }
I20260808 01:09:33.537626 1377 leader_election.cc:302] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a043bcc76cc4126b15e712346791f23; no voters:
I20260808 01:09:33.538751 1377 leader_election.cc:288] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 01:09:33.539227 1380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:33.540745 1380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [term 1 LEADER]: Becoming Leader. State: Replica: 5a043bcc76cc4126b15e712346791f23, State: Running, Role: LEADER
I20260808 01:09:33.541446 1380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a043bcc76cc4126b15e712346791f23" member_type: VOTER }
I20260808 01:09:33.541966 1377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 01:09:33.546489 1382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a043bcc76cc4126b15e712346791f23. Latest consensus state: current_term: 1 leader_uuid: "5a043bcc76cc4126b15e712346791f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a043bcc76cc4126b15e712346791f23" member_type: VOTER } }
I20260808 01:09:33.547245 1382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [sys.catalog]: This master's current role is: LEADER
I20260808 01:09:33.551923 1381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a043bcc76cc4126b15e712346791f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a043bcc76cc4126b15e712346791f23" member_type: VOTER } }
I20260808 01:09:33.553122 1387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 01:09:33.553504 1381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23 [sys.catalog]: This master's current role is: LEADER
I20260808 01:09:33.561429 1387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 01:09:33.562382 32593 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260808 01:09:33.570405 1387 catalog_manager.cc:1347] Generated new cluster ID: 1e49a5b2305d4b958dcbedc89459d597
I20260808 01:09:33.570673 1387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 01:09:33.584389 1387 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 01:09:33.586112 1387 catalog_manager.cc:1504] Loading token signing keys...
I20260808 01:09:33.597596 1387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23: Generated new TSK 0
I20260808 01:09:33.598229 1387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 01:09:33.629580 32593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 01:09:33.635442 1398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:09:33.637195 1399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:09:33.637748 1401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:09:33.638653 32593 server_base.cc:1034] running on GCE node
I20260808 01:09:33.639871 32593 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:09:33.640084 32593 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:09:33.640235 32593 hybrid_clock.cc:648] HybridClock initialized: now 1786151373640222 us; error 0 us; skew 500 ppm
I20260808 01:09:33.640806 32593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:09:33.643352 32593 webserver.cc:460] Webserver started at http://127.31.212.65:35163/ using document root <none> and password file <none>
I20260808 01:09:33.643874 32593 fs_manager.cc:359] Metadata directory not provided
I20260808 01:09:33.644076 32593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:09:33.644343 32593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:09:33.645512 32593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root/instance:
uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
format_stamp: "Formatted at 2026-08-08 01:09:33 on dist-test-slave-mvvj"
I20260808 01:09:33.650012 32593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 01:09:33.653510 1407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:33.654289 32593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 01:09:33.654563 32593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root
uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
format_stamp: "Formatted at 2026-08-08 01:09:33 on dist-test-slave-mvvj"
I20260808 01:09:33.654832 32593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:09:33.663493 32593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 01:09:33.664743 32593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:09:33.666142 32593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 01:09:33.668443 32593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:09:33.668678 32593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:33.668911 32593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 01:09:33.669092 32593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:33.710618 32593 rpc_server.cc:307] RPC server started. Bound to: 127.31.212.65:38291
I20260808 01:09:33.710767 1473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.212.65:38291 every 8 connection(s)
I20260808 01:09:33.715163 32593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 01:09:33.722906 1478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:09:33.725204 1479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:09:33.730293 1482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:09:33.732394 32593 server_base.cc:1034] running on GCE node
I20260808 01:09:33.732249 1474 heartbeater.cc:347] Connected to a master server at 127.31.212.126:33415
I20260808 01:09:33.732951 1474 heartbeater.cc:464] Registering TS with master...
I20260808 01:09:33.733661 1474 heartbeater.cc:511] Master 127.31.212.126:33415 requested a full tablet report, sending...
I20260808 01:09:33.734313 32593 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:09:33.734537 32593 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:09:33.734750 32593 hybrid_clock.cc:648] HybridClock initialized: now 1786151373734731 us; error 0 us; skew 500 ppm
I20260808 01:09:33.735430 32593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:09:33.735899 1342 ts_manager.cc:194] Registered new tserver with Master: 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
I20260808 01:09:33.738133 32593 webserver.cc:460] Webserver started at http://127.31.212.66:43765/ using document root <none> and password file <none>
I20260808 01:09:33.738243 1342 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55752
I20260808 01:09:33.738669 32593 fs_manager.cc:359] Metadata directory not provided
I20260808 01:09:33.738909 32593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:09:33.739185 32593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:09:33.740532 32593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-1-root/instance:
uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
format_stamp: "Formatted at 2026-08-08 01:09:33 on dist-test-slave-mvvj"
I20260808 01:09:33.744879 32593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 01:09:33.748207 1486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:33.748989 32593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 01:09:33.749298 32593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-1-root
uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
format_stamp: "Formatted at 2026-08-08 01:09:33 on dist-test-slave-mvvj"
I20260808 01:09:33.749577 32593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:09:33.768671 32593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 01:09:33.769999 32593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:09:33.771626 32593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 01:09:33.773977 32593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:09:33.774226 32593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:33.774487 32593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 01:09:33.774669 32593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:33.815416 32593 rpc_server.cc:307] RPC server started. Bound to: 127.31.212.66:37093
I20260808 01:09:33.815523 1551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.212.66:37093 every 8 connection(s)
I20260808 01:09:33.820597 32593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 01:09:33.828071 1556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:09:33.829630 1557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:09:33.832212 1560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:09:33.833063 32593 server_base.cc:1034] running on GCE node
I20260808 01:09:33.834060 1553 heartbeater.cc:347] Connected to a master server at 127.31.212.126:33415
I20260808 01:09:33.834164 32593 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:09:33.834425 32593 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:09:33.834462 1553 heartbeater.cc:464] Registering TS with master...
I20260808 01:09:33.834662 32593 hybrid_clock.cc:648] HybridClock initialized: now 1786151373834646 us; error 0 us; skew 500 ppm
I20260808 01:09:33.835326 32593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:09:33.835363 1553 heartbeater.cc:511] Master 127.31.212.126:33415 requested a full tablet report, sending...
I20260808 01:09:33.837857 1342 ts_manager.cc:194] Registered new tserver with Master: 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:33.838248 32593 webserver.cc:460] Webserver started at http://127.31.212.67:43711/ using document root <none> and password file <none>
I20260808 01:09:33.838893 32593 fs_manager.cc:359] Metadata directory not provided
I20260808 01:09:33.839129 32593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:09:33.839464 32593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:09:33.840238 1342 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55756
I20260808 01:09:33.841151 32593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-2-root/instance:
uuid: "a4862d0938c84fc4bd840e4c17acae78"
format_stamp: "Formatted at 2026-08-08 01:09:33 on dist-test-slave-mvvj"
I20260808 01:09:33.846112 32593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260808 01:09:33.849464 1564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:33.850446 32593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 01:09:33.850786 32593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-2-root
uuid: "a4862d0938c84fc4bd840e4c17acae78"
format_stamp: "Formatted at 2026-08-08 01:09:33 on dist-test-slave-mvvj"
I20260808 01:09:33.851131 32593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:09:33.892556 32593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 01:09:33.893884 32593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:09:33.895813 32593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 01:09:33.898458 32593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:09:33.898721 32593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:33.898981 32593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 01:09:33.899151 32593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:33.938920 32593 rpc_server.cc:307] RPC server started. Bound to: 127.31.212.67:35675
I20260808 01:09:33.939033 1629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.212.67:35675 every 8 connection(s)
I20260808 01:09:33.956126 1630 heartbeater.cc:347] Connected to a master server at 127.31.212.126:33415
I20260808 01:09:33.956682 1630 heartbeater.cc:464] Registering TS with master...
I20260808 01:09:33.957449 1630 heartbeater.cc:511] Master 127.31.212.126:33415 requested a full tablet report, sending...
I20260808 01:09:33.959355 1342 ts_manager.cc:194] Registered new tserver with Master: a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:33.960263 32593 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017973087s
I20260808 01:09:33.961535 1342 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55758
I20260808 01:09:33.962970 32593 test_util.cc:268] Using random seed: -970391870
I20260808 01:09:33.990847 1342 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55764:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 01:09:33.997169 1342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 01:09:34.207212 1437 tablet_service.cc:1467] Processing CreateTablet for tablet 516db2422f0644aebfd1999735efdbb8 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260808 01:09:34.208747 1437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 516db2422f0644aebfd1999735efdbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.210263 1429 tablet_service.cc:1467] Processing CreateTablet for tablet 1e59ebd6ca6845789ae50d410b699ea7 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260808 01:09:34.211905 1429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e59ebd6ca6845789ae50d410b699ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.212999 1436 tablet_service.cc:1467] Processing CreateTablet for tablet ec7eb407c2934b7f9db6fa7a3ea0fae5 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260808 01:09:34.214432 1436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7eb407c2934b7f9db6fa7a3ea0fae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.215034 1593 tablet_service.cc:1467] Processing CreateTablet for tablet 516db2422f0644aebfd1999735efdbb8 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260808 01:09:34.216356 1435 tablet_service.cc:1467] Processing CreateTablet for tablet 212d8f7da4b84fae81802fe11cdd4b81 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260808 01:09:34.217701 1435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 212d8f7da4b84fae81802fe11cdd4b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.217578 1592 tablet_service.cc:1467] Processing CreateTablet for tablet ec7eb407c2934b7f9db6fa7a3ea0fae5 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260808 01:09:34.219810 1591 tablet_service.cc:1467] Processing CreateTablet for tablet 212d8f7da4b84fae81802fe11cdd4b81 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260808 01:09:34.221036 1433 tablet_service.cc:1467] Processing CreateTablet for tablet 58547863eabc48eda1552ed4240f68d3 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260808 01:09:34.222298 1433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58547863eabc48eda1552ed4240f68d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.223259 1590 tablet_service.cc:1467] Processing CreateTablet for tablet 87907e719e4c46c4a19015f3f3bba308 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260808 01:09:34.225040 1432 tablet_service.cc:1467] Processing CreateTablet for tablet 260b61a7a7f84a32bda47e79a38ac469 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260808 01:09:34.226418 1432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 260b61a7a7f84a32bda47e79a38ac469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.219666 1434 tablet_service.cc:1467] Processing CreateTablet for tablet 87907e719e4c46c4a19015f3f3bba308 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260808 01:09:34.228199 1434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87907e719e4c46c4a19015f3f3bba308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.229060 1589 tablet_service.cc:1467] Processing CreateTablet for tablet 58547863eabc48eda1552ed4240f68d3 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260808 01:09:34.230556 1428 tablet_service.cc:1467] Processing CreateTablet for tablet bc9cccb55e8444778a9b2833e3d89a5a (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260808 01:09:34.232861 1428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9cccb55e8444778a9b2833e3d89a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.234179 1431 tablet_service.cc:1467] Processing CreateTablet for tablet 36f2393bca80434dbdc089722b52edd1 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260808 01:09:34.234845 1584 tablet_service.cc:1467] Processing CreateTablet for tablet bc9cccb55e8444778a9b2833e3d89a5a (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260808 01:09:34.235525 1431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36f2393bca80434dbdc089722b52edd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.216393 1593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 516db2422f0644aebfd1999735efdbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.237752 1585 tablet_service.cc:1467] Processing CreateTablet for tablet 1e59ebd6ca6845789ae50d410b699ea7 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260808 01:09:34.239109 1585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e59ebd6ca6845789ae50d410b699ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.240128 1588 tablet_service.cc:1467] Processing CreateTablet for tablet 260b61a7a7f84a32bda47e79a38ac469 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260808 01:09:34.232964 1430 tablet_service.cc:1467] Processing CreateTablet for tablet bc8c41cadc374229baf248366012f202 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260808 01:09:34.242494 1430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc8c41cadc374229baf248366012f202. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.232973 1586 tablet_service.cc:1467] Processing CreateTablet for tablet bc8c41cadc374229baf248366012f202 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260808 01:09:34.245079 1586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc8c41cadc374229baf248366012f202. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.231380 1587 tablet_service.cc:1467] Processing CreateTablet for tablet 36f2393bca80434dbdc089722b52edd1 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260808 01:09:34.246862 1587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36f2393bca80434dbdc089722b52edd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.248042 1589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58547863eabc48eda1552ed4240f68d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.251120 1588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 260b61a7a7f84a32bda47e79a38ac469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.256264 1584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9cccb55e8444778a9b2833e3d89a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.260303 1590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87907e719e4c46c4a19015f3f3bba308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.266193 1515 tablet_service.cc:1467] Processing CreateTablet for tablet 516db2422f0644aebfd1999735efdbb8 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260808 01:09:34.267577 1515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 516db2422f0644aebfd1999735efdbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.263713 1591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 212d8f7da4b84fae81802fe11cdd4b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.269085 1514 tablet_service.cc:1467] Processing CreateTablet for tablet ec7eb407c2934b7f9db6fa7a3ea0fae5 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260808 01:09:34.271618 1592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7eb407c2934b7f9db6fa7a3ea0fae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.271631 1513 tablet_service.cc:1467] Processing CreateTablet for tablet 212d8f7da4b84fae81802fe11cdd4b81 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260808 01:09:34.273965 1512 tablet_service.cc:1467] Processing CreateTablet for tablet 87907e719e4c46c4a19015f3f3bba308 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260808 01:09:34.276255 1511 tablet_service.cc:1467] Processing CreateTablet for tablet 58547863eabc48eda1552ed4240f68d3 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260808 01:09:34.278509 1510 tablet_service.cc:1467] Processing CreateTablet for tablet 260b61a7a7f84a32bda47e79a38ac469 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260808 01:09:34.280791 1509 tablet_service.cc:1467] Processing CreateTablet for tablet 36f2393bca80434dbdc089722b52edd1 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260808 01:09:34.282964 1508 tablet_service.cc:1467] Processing CreateTablet for tablet bc8c41cadc374229baf248366012f202 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260808 01:09:34.288626 1507 tablet_service.cc:1467] Processing CreateTablet for tablet 1e59ebd6ca6845789ae50d410b699ea7 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260808 01:09:34.291365 1506 tablet_service.cc:1467] Processing CreateTablet for tablet bc9cccb55e8444778a9b2833e3d89a5a (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260808 01:09:34.292896 1436 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0a7f6aa072473098cb2c51da278a83 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260808 01:09:34.294238 1436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0a7f6aa072473098cb2c51da278a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.299454 1511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58547863eabc48eda1552ed4240f68d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.300889 1514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7eb407c2934b7f9db6fa7a3ea0fae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.302253 1513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 212d8f7da4b84fae81802fe11cdd4b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.303599 1509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36f2393bca80434dbdc089722b52edd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.304960 1508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc8c41cadc374229baf248366012f202. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.306376 1506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9cccb55e8444778a9b2833e3d89a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.307713 1512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87907e719e4c46c4a19015f3f3bba308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.309173 1507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e59ebd6ca6845789ae50d410b699ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.310513 1510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 260b61a7a7f84a32bda47e79a38ac469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.320652 1646 tablet_bootstrap.cc:492] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:34.321478 1587 tablet_service.cc:1467] Processing CreateTablet for tablet 417bc7d1279d48e89c3599b26117aaf9 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260808 01:09:34.322798 1587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417bc7d1279d48e89c3599b26117aaf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.330328 1437 tablet_service.cc:1467] Processing CreateTablet for tablet 417bc7d1279d48e89c3599b26117aaf9 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260808 01:09:34.331712 1437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417bc7d1279d48e89c3599b26117aaf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.340512 1435 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8d1d748c8f4d44942ad2f2e57e9162 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260808 01:09:34.327723 1434 tablet_service.cc:1467] Processing CreateTablet for tablet bcdecb8f50de443095f848f2500fd0a8 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260808 01:09:34.342859 1434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcdecb8f50de443095f848f2500fd0a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.331507 1593 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8d1d748c8f4d44942ad2f2e57e9162 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260808 01:09:34.350287 1435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8d1d748c8f4d44942ad2f2e57e9162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.360638 1646 tablet_bootstrap.cc:654] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:34.374804 1593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8d1d748c8f4d44942ad2f2e57e9162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.408617 1433 tablet_service.cc:1467] Processing CreateTablet for tablet 821d5636d01e4067bac2bbd7475921ef (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260808 01:09:34.409979 1433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 821d5636d01e4067bac2bbd7475921ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.438874 1646 tablet_bootstrap.cc:492] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:34.439567 1646 ts_tablet_manager.cc:1397] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.119s user 0.013s sys 0.010s
I20260808 01:09:34.439630 1587 tablet_service.cc:1467] Processing CreateTablet for tablet 821d5636d01e4067bac2bbd7475921ef (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260808 01:09:34.441118 1587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 821d5636d01e4067bac2bbd7475921ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.444512 1646 raft_consensus.cc:348] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:34.458766 1509 tablet_service.cc:1467] Processing CreateTablet for tablet 417bc7d1279d48e89c3599b26117aaf9 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260808 01:09:34.460690 1431 tablet_service.cc:1467] Processing CreateTablet for tablet 002e3bea322d445e8f5f8f09e2442db4 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260808 01:09:34.462143 1429 tablet_service.cc:1467] Processing CreateTablet for tablet 43bc87d4fdb1436f875364cc78ec3fca (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260808 01:09:34.452176 1646 raft_consensus.cc:374] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:34.479142 1428 tablet_service.cc:1467] Processing CreateTablet for tablet 44f40a08ace944a88ad5bf8ba22f2a40 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260808 01:09:34.483527 1509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417bc7d1279d48e89c3599b26117aaf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.485744 1510 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8d1d748c8f4d44942ad2f2e57e9162 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260808 01:09:34.477504 1646 raft_consensus.cc:729] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:34.488685 1646 consensus_queue.cc:260] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:34.492502 1510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8d1d748c8f4d44942ad2f2e57e9162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.503224 1648 tablet_bootstrap.cc:492] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:34.506302 1592 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0a7f6aa072473098cb2c51da278a83 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260808 01:09:34.512832 1585 tablet_service.cc:1467] Processing CreateTablet for tablet 002e3bea322d445e8f5f8f09e2442db4 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260808 01:09:34.518551 1592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0a7f6aa072473098cb2c51da278a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.523782 1586 tablet_service.cc:1467] Processing CreateTablet for tablet 44f40a08ace944a88ad5bf8ba22f2a40 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260808 01:09:34.519464 1645 tablet_bootstrap.cc:492] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:34.520654 1630 heartbeater.cc:503] Master 127.31.212.126:33415 was elected leader, sending a full tablet report...
I20260808 01:09:34.522887 1431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 002e3bea322d445e8f5f8f09e2442db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.521907 1589 tablet_service.cc:1467] Processing CreateTablet for tablet 43bc87d4fdb1436f875364cc78ec3fca (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260808 01:09:34.491432 1514 tablet_service.cc:1467] Processing CreateTablet for tablet 821d5636d01e4067bac2bbd7475921ef (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260808 01:09:34.527796 1513 tablet_service.cc:1467] Processing CreateTablet for tablet 43bc87d4fdb1436f875364cc78ec3fca (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260808 01:09:34.556384 1646 ts_tablet_manager.cc:1428] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.116s user 0.018s sys 0.013s
I20260808 01:09:34.557096 1506 tablet_service.cc:1467] Processing CreateTablet for tablet 44f40a08ace944a88ad5bf8ba22f2a40 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
W20260808 01:09:34.557493 1390 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:09:34.557542 1646 tablet_bootstrap.cc:492] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:34.530414 1430 tablet_service.cc:1467] Processing CreateTablet for tablet f6bb0551ce74437284392981d2921446 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260808 01:09:34.536801 1436 tablet_service.cc:1467] Processing CreateTablet for tablet 5353a0d85aff48078011ca6efaf9ddfa (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260808 01:09:34.585033 1431 tablet_service.cc:1467] Processing CreateTablet for tablet ea62d4057ca44da4b68e6710052dfbe5 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260808 01:09:34.533608 1586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f40a08ace944a88ad5bf8ba22f2a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.586160 1589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43bc87d4fdb1436f875364cc78ec3fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.536870 1432 tablet_service.cc:1467] Processing CreateTablet for tablet 498a0710a54e491dbd1a79ce64033f62 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260808 01:09:34.588704 1585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 002e3bea322d445e8f5f8f09e2442db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.555172 1645 tablet_bootstrap.cc:654] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:34.537664 1508 tablet_service.cc:1467] Processing CreateTablet for tablet 002e3bea322d445e8f5f8f09e2442db4 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260808 01:09:34.539222 1434 tablet_service.cc:1467] Processing CreateTablet for tablet d80a0f28c7c94d45b7983fc0b1386caf (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260808 01:09:34.542305 1648 tablet_bootstrap.cc:654] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:34.542799 1428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f40a08ace944a88ad5bf8ba22f2a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.549950 1514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 821d5636d01e4067bac2bbd7475921ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.593787 1429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43bc87d4fdb1436f875364cc78ec3fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.595628 1513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43bc87d4fdb1436f875364cc78ec3fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.596150 1434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d80a0f28c7c94d45b7983fc0b1386caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.554666 1587 tablet_service.cc:1467] Processing CreateTablet for tablet f6bb0551ce74437284392981d2921446 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260808 01:09:34.600723 1587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6bb0551ce74437284392981d2921446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.540694 1437 tablet_service.cc:1467] Processing CreateTablet for tablet b316a05b1f7448158ae76530ee0ee127 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260808 01:09:34.569468 1592 tablet_service.cc:1467] Processing CreateTablet for tablet d80a0f28c7c94d45b7983fc0b1386caf (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260808 01:09:34.604486 1508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 002e3bea322d445e8f5f8f09e2442db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.604741 1592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d80a0f28c7c94d45b7983fc0b1386caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.571247 1588 tablet_service.cc:1467] Processing CreateTablet for tablet bcdecb8f50de443095f848f2500fd0a8 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260808 01:09:34.546353 1435 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0008d696f943a3aaa520463af63298 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260808 01:09:34.573719 1591 tablet_service.cc:1467] Processing CreateTablet for tablet 3f18b5317cac4512ad16a64664fab357 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260808 01:09:34.605998 1591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f18b5317cac4512ad16a64664fab357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.607523 1437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b316a05b1f7448158ae76530ee0ee127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.608189 1431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea62d4057ca44da4b68e6710052dfbe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.609122 1435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0008d696f943a3aaa520463af63298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.565143 1515 tablet_service.cc:1467] Processing CreateTablet for tablet f6bb0551ce74437284392981d2921446 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260808 01:09:34.610052 1515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6bb0551ce74437284392981d2921446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.561105 1584 tablet_service.cc:1467] Processing CreateTablet for tablet 498a0710a54e491dbd1a79ce64033f62 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260808 01:09:34.575749 1512 tablet_service.cc:1467] Processing CreateTablet for tablet 498a0710a54e491dbd1a79ce64033f62 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260808 01:09:34.558256 1433 tablet_service.cc:1467] Processing CreateTablet for tablet 3f18b5317cac4512ad16a64664fab357 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260808 01:09:34.577565 1590 tablet_service.cc:1467] Processing CreateTablet for tablet b316a05b1f7448158ae76530ee0ee127 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260808 01:09:34.584048 1593 tablet_service.cc:1467] Processing CreateTablet for tablet 5353a0d85aff48078011ca6efaf9ddfa (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260808 01:09:34.532723 1511 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0a7f6aa072473098cb2c51da278a83 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260808 01:09:34.612157 1432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 498a0710a54e491dbd1a79ce64033f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.628280 1591 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0008d696f943a3aaa520463af63298 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260808 01:09:34.612701 1588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcdecb8f50de443095f848f2500fd0a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.614548 1645 tablet_bootstrap.cc:492] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:34.614951 1646 tablet_bootstrap.cc:654] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:34.574064 1507 tablet_service.cc:1467] Processing CreateTablet for tablet 5353a0d85aff48078011ca6efaf9ddfa (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260808 01:09:34.636159 1584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 498a0710a54e491dbd1a79ce64033f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.616699 1648 tablet_bootstrap.cc:492] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:34.636914 1648 ts_tablet_manager.cc:1397] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.134s user 0.004s sys 0.009s
I20260808 01:09:34.637486 1590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b316a05b1f7448158ae76530ee0ee127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.626564 1586 tablet_service.cc:1467] Processing CreateTablet for tablet ea62d4057ca44da4b68e6710052dfbe5 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260808 01:09:34.640198 1591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0008d696f943a3aaa520463af63298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.640296 1433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f18b5317cac4512ad16a64664fab357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.639734 1648 raft_consensus.cc:348] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:34.641965 1586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea62d4057ca44da4b68e6710052dfbe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.642566 1436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5353a0d85aff48078011ca6efaf9ddfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.636222 1645 ts_tablet_manager.cc:1397] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.144s user 0.013s sys 0.000s
I20260808 01:09:34.644637 1431 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad0577278e046ad8387388b9e5e98a0 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260808 01:09:34.646078 1431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad0577278e046ad8387388b9e5e98a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.641247 1648 raft_consensus.cc:374] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:34.648126 1648 raft_consensus.cc:729] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:34.649030 1648 consensus_queue.cc:260] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:34.647589 1645 raft_consensus.cc:348] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:34.649525 1430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6bb0551ce74437284392981d2921446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.654278 1645 raft_consensus.cc:374] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:34.672364 1645 raft_consensus.cc:729] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:34.694504 1509 tablet_service.cc:1467] Processing CreateTablet for tablet d80a0f28c7c94d45b7983fc0b1386caf (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260808 01:09:34.697438 1586 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad0577278e046ad8387388b9e5e98a0 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260808 01:09:34.704159 1645 consensus_queue.cc:260] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:34.679934 1431 tablet_service.cc:1467] Processing CreateTablet for tablet bc54c3f2f5e04f63b88243c0d2d6babe (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260808 01:09:34.707597 1553 heartbeater.cc:503] Master 127.31.212.126:33415 was elected leader, sending a full tablet report...
I20260808 01:09:34.711205 1648 ts_tablet_manager.cc:1428] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.074s user 0.012s sys 0.000s
I20260808 01:09:34.684396 1506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f40a08ace944a88ad5bf8ba22f2a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.713645 1593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5353a0d85aff48078011ca6efaf9ddfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.720693 1431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc54c3f2f5e04f63b88243c0d2d6babe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.724619 1512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 498a0710a54e491dbd1a79ce64033f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.738870 1645 ts_tablet_manager.cc:1428] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.094s user 0.007s sys 0.014s
I20260808 01:09:34.739907 1474 heartbeater.cc:503] Master 127.31.212.126:33415 was elected leader, sending a full tablet report...
I20260808 01:09:34.740494 1645 tablet_bootstrap.cc:492] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:34.745469 1648 tablet_bootstrap.cc:492] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:34.752068 1645 tablet_bootstrap.cc:654] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:34.757342 1512 tablet_service.cc:1467] Processing CreateTablet for tablet bcdecb8f50de443095f848f2500fd0a8 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260808 01:09:34.759122 1429 tablet_service.cc:1467] Processing CreateTablet for tablet c12c84f631ee4ca7b405e28317db1b8a (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260808 01:09:34.755332 1507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5353a0d85aff48078011ca6efaf9ddfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.767113 1509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d80a0f28c7c94d45b7983fc0b1386caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.770466 1511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0a7f6aa072473098cb2c51da278a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.777153 1428 tablet_service.cc:1467] Processing CreateTablet for tablet 921f3eb5150b4105a4b68355cca8cb7a (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260808 01:09:34.779232 1428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921f3eb5150b4105a4b68355cca8cb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.705446 1437 tablet_service.cc:1467] Processing CreateTablet for tablet 45ac77bfbd0c4849bbda5cd2e754a927 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260808 01:09:34.759207 1510 tablet_service.cc:1467] Processing CreateTablet for tablet 3f18b5317cac4512ad16a64664fab357 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260808 01:09:34.790526 1513 tablet_service.cc:1467] Processing CreateTablet for tablet b316a05b1f7448158ae76530ee0ee127 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260808 01:09:34.791337 1645 tablet_bootstrap.cc:492] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:34.791661 1429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12c84f631ee4ca7b405e28317db1b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.791898 1586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad0577278e046ad8387388b9e5e98a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.796425 1587 tablet_service.cc:1467] Processing CreateTablet for tablet c12c84f631ee4ca7b405e28317db1b8a (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260808 01:09:34.800521 1512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcdecb8f50de443095f848f2500fd0a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.803339 1508 tablet_service.cc:1467] Processing CreateTablet for tablet ea62d4057ca44da4b68e6710052dfbe5 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260808 01:09:34.804190 1648 tablet_bootstrap.cc:654] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:34.815052 1513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b316a05b1f7448158ae76530ee0ee127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.815773 1587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12c84f631ee4ca7b405e28317db1b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.816076 1592 tablet_service.cc:1467] Processing CreateTablet for tablet bc54c3f2f5e04f63b88243c0d2d6babe (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260808 01:09:34.817420 1592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc54c3f2f5e04f63b88243c0d2d6babe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.817956 1510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f18b5317cac4512ad16a64664fab357. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:09:34.819994 1554 tablet.cc:2369] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:09:34.810647 1645 ts_tablet_manager.cc:1397] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.071s user 0.015s sys 0.025s
I20260808 01:09:34.826588 1645 raft_consensus.cc:348] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:34.827401 1645 raft_consensus.cc:374] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:34.828474 1437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ac77bfbd0c4849bbda5cd2e754a927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.735728 1590 tablet_service.cc:1467] Processing CreateTablet for tablet 45ac77bfbd0c4849bbda5cd2e754a927 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260808 01:09:34.836585 1508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea62d4057ca44da4b68e6710052dfbe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.837024 1589 tablet_service.cc:1467] Processing CreateTablet for tablet 921f3eb5150b4105a4b68355cca8cb7a (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260808 01:09:34.838054 1585 tablet_service.cc:1467] Processing CreateTablet for tablet d6b8d42642b84d3f9a2adbbaf3fd5417 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260808 01:09:34.855368 1591 tablet_service.cc:1467] Processing CreateTablet for tablet 0108cd20b4bb4957b8a59eebd68479e4 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260808 01:09:34.836606 1590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ac77bfbd0c4849bbda5cd2e754a927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.871438 1513 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad0577278e046ad8387388b9e5e98a0 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260808 01:09:34.872790 1513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad0577278e046ad8387388b9e5e98a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.873445 1648 tablet_bootstrap.cc:492] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:34.873962 1648 ts_tablet_manager.cc:1397] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.129s user 0.006s sys 0.014s
I20260808 01:09:34.875389 1584 tablet_service.cc:1467] Processing CreateTablet for tablet ca9890c6b79a40fcbbf70e44142a95d0 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260808 01:09:34.876732 1648 raft_consensus.cc:348] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:34.877141 1510 tablet_service.cc:1467] Processing CreateTablet for tablet bc54c3f2f5e04f63b88243c0d2d6babe (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260808 01:09:34.880205 1589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921f3eb5150b4105a4b68355cca8cb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.827688 1645 raft_consensus.cc:729] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:34.892398 1645 consensus_queue.cc:260] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:34.882458 1648 raft_consensus.cc:374] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:34.893528 1648 raft_consensus.cc:729] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:34.895687 1645 ts_tablet_manager.cc:1428] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.072s user 0.005s sys 0.001s
I20260808 01:09:34.896701 1645 tablet_bootstrap.cc:492] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:34.897416 1646 tablet_bootstrap.cc:492] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:34.897935 1646 ts_tablet_manager.cc:1397] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.341s user 0.012s sys 0.022s
I20260808 01:09:34.900147 1648 consensus_queue.cc:260] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:34.901997 1512 tablet_service.cc:1467] Processing CreateTablet for tablet 45ac77bfbd0c4849bbda5cd2e754a927 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260808 01:09:34.902567 1648 ts_tablet_manager.cc:1428] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.028s user 0.006s sys 0.002s
I20260808 01:09:34.907183 1435 tablet_service.cc:1467] Processing CreateTablet for tablet ca9890c6b79a40fcbbf70e44142a95d0 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260808 01:09:34.910439 1510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc54c3f2f5e04f63b88243c0d2d6babe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.910653 1435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca9890c6b79a40fcbbf70e44142a95d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.912827 1648 tablet_bootstrap.cc:492] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:34.916127 1591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0108cd20b4bb4957b8a59eebd68479e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.920766 1513 tablet_service.cc:1467] Processing CreateTablet for tablet c12c84f631ee4ca7b405e28317db1b8a (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260808 01:09:34.922116 1513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12c84f631ee4ca7b405e28317db1b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.924860 1430 tablet_service.cc:1467] Processing CreateTablet for tablet 0108cd20b4bb4957b8a59eebd68479e4 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260808 01:09:34.925372 1645 tablet_bootstrap.cc:654] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:34.926185 1511 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0008d696f943a3aaa520463af63298 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260808 01:09:34.927479 1511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0008d696f943a3aaa520463af63298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.929982 1509 tablet_service.cc:1467] Processing CreateTablet for tablet 921f3eb5150b4105a4b68355cca8cb7a (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260808 01:09:34.931227 1509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921f3eb5150b4105a4b68355cca8cb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.926198 1430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0108cd20b4bb4957b8a59eebd68479e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.934124 1434 tablet_service.cc:1467] Processing CreateTablet for tablet 82bb58b6f7c14c22814cfac6d2b2bf6d (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260808 01:09:34.935559 1648 tablet_bootstrap.cc:654] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:34.935946 1512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ac77bfbd0c4849bbda5cd2e754a927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.941184 1587 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2611aecd26485e8912ab55890ec0fa (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260808 01:09:34.967607 1654 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:34.968144 1654 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:34.943481 1646 raft_consensus.cc:348] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:34.970321 1646 raft_consensus.cc:374] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:34.970739 1654 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:34.970935 1646 raft_consensus.cc:729] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:34.971616 1649 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:34.972102 1649 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:34.972584 1646 consensus_queue.cc:260] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:34.975209 1649 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:34.975124 1433 tablet_service.cc:1467] Processing CreateTablet for tablet d6b8d42642b84d3f9a2adbbaf3fd5417 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260808 01:09:34.950402 1514 tablet_service.cc:1467] Processing CreateTablet for tablet d6b8d42642b84d3f9a2adbbaf3fd5417 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260808 01:09:34.951673 1585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b8d42642b84d3f9a2adbbaf3fd5417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.958014 1434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82bb58b6f7c14c22814cfac6d2b2bf6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:34.941066 1436 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2611aecd26485e8912ab55890ec0fa (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260808 01:09:35.002565 1587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2611aecd26485e8912ab55890ec0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.005309 1432 tablet_service.cc:1467] Processing CreateTablet for tablet 846d38a10c5846789f0b926b61905c0b (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260808 01:09:35.006615 1433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b8d42642b84d3f9a2adbbaf3fd5417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.006441 1590 tablet_service.cc:1467] Processing CreateTablet for tablet 82bb58b6f7c14c22814cfac6d2b2bf6d (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260808 01:09:35.007683 1590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82bb58b6f7c14c22814cfac6d2b2bf6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.012446 1584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca9890c6b79a40fcbbf70e44142a95d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.019115 1432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 846d38a10c5846789f0b926b61905c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.022372 1436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2611aecd26485e8912ab55890ec0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.020596 1514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b8d42642b84d3f9a2adbbaf3fd5417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.029155 1645 tablet_bootstrap.cc:492] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:35.029642 1645 ts_tablet_manager.cc:1397] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.133s user 0.019s sys 0.014s
I20260808 01:09:35.034319 1428 tablet_service.cc:1467] Processing CreateTablet for tablet e354fdfbf2264912b39cd20be2cb19f8 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260808 01:09:35.035597 1428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e354fdfbf2264912b39cd20be2cb19f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.082093 1507 tablet_service.cc:1467] Processing CreateTablet for tablet 0108cd20b4bb4957b8a59eebd68479e4 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260808 01:09:35.085369 1431 tablet_service.cc:1467] Processing CreateTablet for tablet efd5468b2c6e4509ae17a98f9509ff03 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260808 01:09:35.031255 1586 tablet_service.cc:1467] Processing CreateTablet for tablet e354fdfbf2264912b39cd20be2cb19f8 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260808 01:09:35.088945 1593 tablet_service.cc:1467] Processing CreateTablet for tablet 846d38a10c5846789f0b926b61905c0b (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260808 01:09:35.046536 1508 tablet_service.cc:1467] Processing CreateTablet for tablet ca9890c6b79a40fcbbf70e44142a95d0 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260808 01:09:35.092420 1586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e354fdfbf2264912b39cd20be2cb19f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.093567 1588 tablet_service.cc:1467] Processing CreateTablet for tablet efd5468b2c6e4509ae17a98f9509ff03 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260808 01:09:35.092461 1507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0108cd20b4bb4957b8a59eebd68479e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.092101 1589 tablet_service.cc:1467] Processing CreateTablet for tablet 55fb799d94774eb18fed4dd8fdb5ac63 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260808 01:09:35.100446 1593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 846d38a10c5846789f0b926b61905c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.102962 1510 tablet_service.cc:1467] Processing CreateTablet for tablet 82bb58b6f7c14c22814cfac6d2b2bf6d (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260808 01:09:35.107110 1437 tablet_service.cc:1467] Processing CreateTablet for tablet 55fb799d94774eb18fed4dd8fdb5ac63 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260808 01:09:35.087312 1431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efd5468b2c6e4509ae17a98f9509ff03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.035792 1646 ts_tablet_manager.cc:1428] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.138s user 0.010s sys 0.000s
I20260808 01:09:35.127307 1587 tablet_service.cc:1467] Processing CreateTablet for tablet 1355c134ca5f45b381fb2b12b1865adb (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260808 01:09:35.128684 1587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1355c134ca5f45b381fb2b12b1865adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.130800 1652 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:35.133513 1648 tablet_bootstrap.cc:492] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:35.134033 1648 ts_tablet_manager.cc:1397] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.221s user 0.004s sys 0.025s
I20260808 01:09:35.134925 1508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca9890c6b79a40fcbbf70e44142a95d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.128564 1429 tablet_service.cc:1467] Processing CreateTablet for tablet 58c8c7709e68463182371aa8224d2da2 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260808 01:09:35.139720 1511 tablet_service.cc:1467] Processing CreateTablet for tablet e354fdfbf2264912b39cd20be2cb19f8 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260808 01:09:35.141147 1511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e354fdfbf2264912b39cd20be2cb19f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.131283 1652 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.152179 1588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efd5468b2c6e4509ae17a98f9509ff03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.153019 1433 tablet_service.cc:1467] Processing CreateTablet for tablet a19c1759594d4722abadf576352f9a98 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260808 01:09:35.154574 1510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82bb58b6f7c14c22814cfac6d2b2bf6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.063596 1506 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2611aecd26485e8912ab55890ec0fa (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260808 01:09:35.163722 1652 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:35.164283 1652 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:35.164691 1652 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.165711 1585 tablet_service.cc:1467] Processing CreateTablet for tablet 58c8c7709e68463182371aa8224d2da2 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260808 01:09:35.166507 1652 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:35.163100 1589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fb799d94774eb18fed4dd8fdb5ac63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.168648 1515 tablet_service.cc:1467] Processing CreateTablet for tablet 846d38a10c5846789f0b926b61905c0b (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260808 01:09:35.169970 1515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 846d38a10c5846789f0b926b61905c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.174961 1590 tablet_service.cc:1467] Processing CreateTablet for tablet a19c1759594d4722abadf576352f9a98 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260808 01:09:35.191155 1585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c8c7709e68463182371aa8224d2da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.198774 1430 tablet_service.cc:1467] Processing CreateTablet for tablet eba23837e3184c0d99703d8b5b0accbe (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260808 01:09:35.210355 1590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19c1759594d4722abadf576352f9a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.185222 1437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fb799d94774eb18fed4dd8fdb5ac63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.212093 1429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c8c7709e68463182371aa8224d2da2. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:09:35.195928 1328 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 47233.4us
I20260808 01:09:35.212842 1433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19c1759594d4722abadf576352f9a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.213028 1506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2611aecd26485e8912ab55890ec0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.170636 1435 tablet_service.cc:1467] Processing CreateTablet for tablet 1355c134ca5f45b381fb2b12b1865adb (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260808 01:09:35.214747 1430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba23837e3184c0d99703d8b5b0accbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.215512 1435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1355c134ca5f45b381fb2b12b1865adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.220027 1512 tablet_service.cc:1467] Processing CreateTablet for tablet efd5468b2c6e4509ae17a98f9509ff03 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260808 01:09:35.220929 1654 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:35.221294 1512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efd5468b2c6e4509ae17a98f9509ff03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.221395 1654 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.223417 1654 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:35.183326 1646 tablet_bootstrap.cc:492] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:35.229723 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:35.230108 1648 raft_consensus.cc:348] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.230851 1648 raft_consensus.cc:374] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:35.231135 1648 raft_consensus.cc:729] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:35.231271 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:35.149012 1645 raft_consensus.cc:348] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.232152 1645 raft_consensus.cc:374] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:35.232470 1645 raft_consensus.cc:729] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
W20260808 01:09:35.234180 1410 leader_election.cc:341] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:09:35.235018 1410 leader_election.cc:341] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:35.233579 1648 consensus_queue.cc:260] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.242158 1648 ts_tablet_manager.cc:1428] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.108s user 0.005s sys 0.002s
I20260808 01:09:35.244047 1648 tablet_bootstrap.cc:492] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:35.253742 1513 tablet_service.cc:1467] Processing CreateTablet for tablet 55fb799d94774eb18fed4dd8fdb5ac63 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260808 01:09:35.255259 1513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fb799d94774eb18fed4dd8fdb5ac63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.265802 1646 tablet_bootstrap.cc:654] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:35.233163 1645 consensus_queue.cc:260] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.243492 1586 tablet_service.cc:1467] Processing CreateTablet for tablet eba23837e3184c0d99703d8b5b0accbe (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260808 01:09:35.267766 1586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba23837e3184c0d99703d8b5b0accbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.268544 1645 ts_tablet_manager.cc:1428] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.239s user 0.006s sys 0.003s
I20260808 01:09:35.271420 1649 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:35.271922 1649 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.273883 1649 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:35.278398 1645 tablet_bootstrap.cc:492] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:35.279567 1587 tablet_service.cc:1467] Processing CreateTablet for tablet 6b1ad14dc361495f8e43a49a35526563 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260808 01:09:35.289891 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:35.301940 1587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b1ad14dc361495f8e43a49a35526563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.302485 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:09:35.304447 1568 leader_election.cc:341] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:09:35.305498 1568 leader_election.cc:341] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:35.307161 1584 tablet_service.cc:1467] Processing CreateTablet for tablet 93b19a284f2b41fe843e33e399ea8ea7 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260808 01:09:35.325721 1431 tablet_service.cc:1467] Processing CreateTablet for tablet 73a11329d12c4ba4abd2b4e0e29816c7 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260808 01:09:35.327116 1431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a11329d12c4ba4abd2b4e0e29816c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.327116 1584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b19a284f2b41fe843e33e399ea8ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.331254 1591 tablet_service.cc:1467] Processing CreateTablet for tablet 73a11329d12c4ba4abd2b4e0e29816c7 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260808 01:09:35.335135 1509 tablet_service.cc:1467] Processing CreateTablet for tablet 1355c134ca5f45b381fb2b12b1865adb (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260808 01:09:35.336768 1509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1355c134ca5f45b381fb2b12b1865adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.348767 1590 tablet_service.cc:1467] Processing CreateTablet for tablet a83a963773094d6db6495b7009fd8690 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260808 01:09:35.360682 1648 tablet_bootstrap.cc:654] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:35.381642 1590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83a963773094d6db6495b7009fd8690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.382139 1436 tablet_service.cc:1467] Processing CreateTablet for tablet 93b19a284f2b41fe843e33e399ea8ea7 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260808 01:09:35.412151 1436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b19a284f2b41fe843e33e399ea8ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.408900 1645 tablet_bootstrap.cc:654] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:35.423254 1429 tablet_service.cc:1467] Processing CreateTablet for tablet 6b1ad14dc361495f8e43a49a35526563 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260808 01:09:35.370515 1514 tablet_service.cc:1467] Processing CreateTablet for tablet 58c8c7709e68463182371aa8224d2da2 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260808 01:09:35.425038 1507 tablet_service.cc:1467] Processing CreateTablet for tablet a19c1759594d4722abadf576352f9a98 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260808 01:09:35.425951 1514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c8c7709e68463182371aa8224d2da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.429102 1591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a11329d12c4ba4abd2b4e0e29816c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.425038 1429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b1ad14dc361495f8e43a49a35526563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.420140 1589 tablet_service.cc:1467] Processing CreateTablet for tablet 704a8e4165cc49079ada0c7993c21187 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260808 01:09:35.435890 1589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 704a8e4165cc49079ada0c7993c21187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.445004 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:35.445744 1524 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:35.447445 1567 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:09:35.448680 1649 raft_consensus.cc:2781] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:35.450950 1507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19c1759594d4722abadf576352f9a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.449041 1649 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:35.464092 1649 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:35.452759 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:35.453868 1508 tablet_service.cc:1467] Processing CreateTablet for tablet eba23837e3184c0d99703d8b5b0accbe (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260808 01:09:35.454046 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:35.454085 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:35.458083 1587 tablet_service.cc:1467] Processing CreateTablet for tablet a539e94f6f744f20b65ea3fa4bbf8be4 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
W20260808 01:09:35.475386 1489 leader_election.cc:341] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:35.476779 1587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a539e94f6f744f20b65ea3fa4bbf8be4. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:09:35.479724 1567 leader_election.cc:341] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:35.487198 1567 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:35.487289 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:35.488705 1508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba23837e3184c0d99703d8b5b0accbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.488751 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:09:35.491024 1490 leader_election.cc:341] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:35.491642 1490 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:35.492806 1668 raft_consensus.cc:2726] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:09:35.494177 1490 leader_election.cc:341] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:35.496621 1648 tablet_bootstrap.cc:492] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:35.504248 1648 ts_tablet_manager.cc:1397] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.260s user 0.010s sys 0.015s
I20260808 01:09:35.505514 1435 tablet_service.cc:1467] Processing CreateTablet for tablet 704a8e4165cc49079ada0c7993c21187 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260808 01:09:35.506968 1435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 704a8e4165cc49079ada0c7993c21187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.482648 1585 tablet_service.cc:1467] Processing CreateTablet for tablet d9e2d40d6b2943d883579bc4571ed4c1 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260808 01:09:35.510603 1515 tablet_service.cc:1467] Processing CreateTablet for tablet 73a11329d12c4ba4abd2b4e0e29816c7 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260808 01:09:35.511895 1434 tablet_service.cc:1467] Processing CreateTablet for tablet 31f1af2e7c504c919e34f12663452f2a (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260808 01:09:35.513326 1434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f1af2e7c504c919e34f12663452f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.501068 1432 tablet_service.cc:1467] Processing CreateTablet for tablet a539e94f6f744f20b65ea3fa4bbf8be4 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260808 01:09:35.502787 1592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:45612 (request call id 27) took 1094 ms. Trace:
I20260808 01:09:35.515358 1506 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58250 (request call id 34) took 1084 ms. Trace:
I20260808 01:09:35.516309 1585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e2d40d6b2943d883579bc4571ed4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.523489 1654 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:35.484820 1428 tablet_service.cc:1467] Processing CreateTablet for tablet a83a963773094d6db6495b7009fd8690 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260808 01:09:35.524091 1654 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.526467 1654 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:35.496235 1593 tablet_service.cc:1467] Processing CreateTablet for tablet 31f1af2e7c504c919e34f12663452f2a (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260808 01:09:35.528880 1593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f1af2e7c504c919e34f12663452f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.523540 1592 rpcz_store.cc:276] 0808 01:09:34.408591 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:34.730836 (+322245us) service_pool.cc:227] Handling call
0808 01:09:34.816983 (+ 86147us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:34.817079 (+ 96us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.040235 (+223156us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.475332 (+435097us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.502717 (+ 27385us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:09:35.530241 1646 tablet_bootstrap.cc:492] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:35.530766 1646 ts_tablet_manager.cc:1397] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.373s user 0.023s sys 0.032s
I20260808 01:09:35.530652 1592 tablet_service.cc:1467] Processing CreateTablet for tablet 1ccc57605ae24bf4bacee4e4741bbe95 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260808 01:09:35.532277 1592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccc57605ae24bf4bacee4e4741bbe95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.503574 1649 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.534894 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:35.537201 1603 raft_consensus.cc:2379] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 1: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:09:35.538535 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:09:35.540392 1568 leader_election.cc:341] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:35.541245 1489 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:35.541085 1646 raft_consensus.cc:348] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.541918 1646 raft_consensus.cc:374] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:35.542261 1646 raft_consensus.cc:729] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:35.542575 1668 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:35.543781 1513 tablet_service.cc:1467] Processing CreateTablet for tablet 93b19a284f2b41fe843e33e399ea8ea7 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
W20260808 01:09:35.545106 1410 leader_election.cc:341] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:35.547205 1428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83a963773094d6db6495b7009fd8690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.547708 1646 consensus_queue.cc:260] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.549170 1645 tablet_bootstrap.cc:492] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:35.549685 1645 ts_tablet_manager.cc:1397] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.272s user 0.007s sys 0.012s
I20260808 01:09:35.551164 1591 tablet_service.cc:1467] Processing CreateTablet for tablet 9ea0da159669492a90fc257d91f4ba5c (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260808 01:09:35.552736 1591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ea0da159669492a90fc257d91f4ba5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.553099 1645 raft_consensus.cc:348] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.554003 1645 raft_consensus.cc:374] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:35.554334 1645 raft_consensus.cc:729] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:35.535940 1515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a11329d12c4ba4abd2b4e0e29816c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.507768 1648 raft_consensus.cc:348] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.542209 1512 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58250 (request call id 38) took 1006 ms. Trace:
I20260808 01:09:35.568305 1649 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:35.541936 1433 tablet_service.cc:1467] Processing CreateTablet for tablet 9ea0da159669492a90fc257d91f4ba5c (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260808 01:09:35.564682 1675 raft_consensus.cc:2726] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:09:35.566807 1390 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:09:35.567956 1645 consensus_queue.cc:260] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.574612 1588 tablet_service.cc:1467] Processing CreateTablet for tablet b3583c46ad2f4aa8875c06ef46cde18a (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260808 01:09:35.575819 1645 ts_tablet_manager.cc:1428] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.026s user 0.004s sys 0.002s
I20260808 01:09:35.576172 1588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3583c46ad2f4aa8875c06ef46cde18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.577059 1645 tablet_bootstrap.cc:492] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:35.580762 1437 tablet_service.cc:1467] Processing CreateTablet for tablet d9e2d40d6b2943d883579bc4571ed4c1 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260808 01:09:35.563308 1648 raft_consensus.cc:374] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:35.584535 1648 raft_consensus.cc:729] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:35.559679 1510 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58250 (request call id 35) took 1079 ms. Trace:
I20260808 01:09:35.585427 1510 rpcz_store.cc:276] 0808 01:09:34.478808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.101958 (+623150us) service_pool.cc:227] Handling call
0808 01:09:35.121916 (+ 19958us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.130110 (+ 8194us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.393005 (+262895us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.504160 (+111155us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.558551 (+ 54391us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:09:35.586133 1430 tablet_service.cc:1467] Processing CreateTablet for tablet b3583c46ad2f4aa8875c06ef46cde18a (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
W20260808 01:09:35.586259 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 106583us
I20260808 01:09:35.587337 1510 tablet_service.cc:1467] Processing CreateTablet for tablet 6b1ad14dc361495f8e43a49a35526563 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260808 01:09:35.533062 1506 rpcz_store.cc:276] 0808 01:09:34.431168 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.062675 (+631507us) service_pool.cc:227] Handling call
0808 01:09:35.187894 (+125219us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.212562 (+ 24668us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.370244 (+157682us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.514804 (+144560us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.515296 (+ 492us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:09:35.596122 1432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a539e94f6f744f20b65ea3fa4bbf8be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.595573 1513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b19a284f2b41fe843e33e399ea8ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.567938 1512 rpcz_store.cc:276] 0808 01:09:34.536084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.219097 (+683013us) service_pool.cc:227] Handling call
0808 01:09:35.220841 (+ 1744us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.220935 (+ 94us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.368584 (+147649us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.541647 (+173063us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.542148 (+ 501us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:09:35.608239 1433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ea0da159669492a90fc257d91f4ba5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.610374 1512 tablet_service.cc:1467] Processing CreateTablet for tablet 704a8e4165cc49079ada0c7993c21187 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260808 01:09:35.617017 1512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 704a8e4165cc49079ada0c7993c21187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.607692 1506 tablet_service.cc:1467] Processing CreateTablet for tablet a83a963773094d6db6495b7009fd8690 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260808 01:09:35.623790 1506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83a963773094d6db6495b7009fd8690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.616842 1590 tablet_service.cc:1467] Processing CreateTablet for tablet a3ce33a519294ae1bce66c8aca8cf601 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260808 01:09:35.612321 1437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e2d40d6b2943d883579bc4571ed4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.626030 1590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ce33a519294ae1bce66c8aca8cf601. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.596560 1584 tablet_service.cc:1467] Processing CreateTablet for tablet a27b15da83cb42d49d1b0a703b5955ea (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260808 01:09:35.641351 1652 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:35.641713 1584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a27b15da83cb42d49d1b0a703b5955ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.641930 1652 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:35.642692 1328 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 55332.5us
I20260808 01:09:35.646834 1648 consensus_queue.cc:260] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.647732 1646 ts_tablet_manager.cc:1428] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.117s user 0.011s sys 0.026s
I20260808 01:09:35.648876 1646 tablet_bootstrap.cc:492] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:35.651526 1510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b1ad14dc361495f8e43a49a35526563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.647624 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:35.651902 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:09:35.654088 1490 leader_election.cc:341] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:35.641410 1430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3583c46ad2f4aa8875c06ef46cde18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.664021 1652 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:35.665251 1668 raft_consensus.cc:2726] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:35.666046 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:09:35.667923 1489 leader_election.cc:341] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:35.668388 1489 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:35.669620 1652 raft_consensus.cc:2726] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:35.676678 1509 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58250 (request call id 40) took 1099 ms. Trace:
I20260808 01:09:35.677148 1509 rpcz_store.cc:276] 0808 01:09:34.577738 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.334133 (+756395us) service_pool.cc:227] Handling call
0808 01:09:35.336240 (+ 2107us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.336359 (+ 119us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.481075 (+144716us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.676090 (+195015us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.676611 (+ 521us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:09:35.681375 1524 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:35.682331 1514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58250 (request call id 41) took 1096 ms. Trace:
I20260808 01:09:35.682753 1514 rpcz_store.cc:276] 0808 01:09:34.586409 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.362718 (+776309us) service_pool.cc:227] Handling call
0808 01:09:35.425403 (+ 62685us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.425505 (+ 102us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.524461 (+ 98956us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.681746 (+157285us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.682268 (+ 522us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:09:35.684021 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:35.685644 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:35.687136 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:09:35.688364 1410 leader_election.cc:341] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:35.688814 1410 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:35.688956 1586 tablet_service.cc:1467] Processing CreateTablet for tablet 2355fe5dfc824ec78c044695be9d32de (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260808 01:09:35.690006 1654 raft_consensus.cc:2726] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:09:35.690954 1410 leader_election.cc:341] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:35.693041 1511 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58250 (request call id 37) took 1213 ms. Trace:
I20260808 01:09:35.679194 1509 tablet_service.cc:1467] Processing CreateTablet for tablet a539e94f6f744f20b65ea3fa4bbf8be4 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260808 01:09:35.702154 1410 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:35.704180 1654 raft_consensus.cc:2726] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:09:35.704702 1410 leader_election.cc:341] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:35.703713 1587 tablet_service.cc:1467] Processing CreateTablet for tablet a668c2efd2dd4f5b946b70904c143c17 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260808 01:09:35.705173 1410 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:35.705943 1654 raft_consensus.cc:2726] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:35.706754 1587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a668c2efd2dd4f5b946b70904c143c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.699107 1511 rpcz_store.cc:276] 0808 01:09:34.479688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.138796 (+659108us) service_pool.cc:227] Handling call
0808 01:09:35.140656 (+ 1860us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.140748 (+ 92us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.508093 (+367345us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.692500 (+184407us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.692961 (+ 461us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:09:35.718104 1511 tablet_service.cc:1467] Processing CreateTablet for tablet d9e2d40d6b2943d883579bc4571ed4c1 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260808 01:09:35.715747 1646 tablet_bootstrap.cc:654] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:35.699642 1567 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:09:35.731616 1509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a539e94f6f744f20b65ea3fa4bbf8be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.735662 1507 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58250 (request call id 42) took 1057 ms. Trace:
I20260808 01:09:35.736156 1507 rpcz_store.cc:276] 0808 01:09:34.686541 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.424040 (+737499us) service_pool.cc:227] Handling call
0808 01:09:35.429041 (+ 5001us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.429154 (+ 113us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.549772 (+120618us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.719364 (+169592us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.735590 (+ 16226us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20946}
I20260808 01:09:35.739346 1507 tablet_service.cc:1467] Processing CreateTablet for tablet 9ea0da159669492a90fc257d91f4ba5c (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260808 01:09:35.740753 1507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ea0da159669492a90fc257d91f4ba5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.732753 1586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2355fe5dfc824ec78c044695be9d32de. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:09:35.750573 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 58788.2us
I20260808 01:09:35.750389 1514 tablet_service.cc:1467] Processing CreateTablet for tablet 31f1af2e7c504c919e34f12663452f2a (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260808 01:09:35.754279 1431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55036 (request call id 44) took 1061 ms. Trace:
I20260808 01:09:35.775682 1428 tablet_service.cc:1467] Processing CreateTablet for tablet a3ce33a519294ae1bce66c8aca8cf601 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260808 01:09:35.777060 1428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ce33a519294ae1bce66c8aca8cf601. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.776268 1431 rpcz_store.cc:276] 0808 01:09:34.693115 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.320350 (+627235us) service_pool.cc:227] Handling call
0808 01:09:35.326553 (+ 6203us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.326669 (+ 116us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.525874 (+199205us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.753732 (+227858us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.754218 (+ 486us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:09:35.778014 1649 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:35.768661 1648 ts_tablet_manager.cc:1428] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.264s user 0.014s sys 0.002s
I20260808 01:09:35.770366 1645 tablet_bootstrap.cc:654] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:35.748802 1676 raft_consensus.cc:2781] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:35.773795 1436 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55036 (request call id 45) took 1038 ms. Trace:
I20260808 01:09:35.778512 1649 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.782245 1676 raft_consensus.cc:686] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:35.782799 1649 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:35.783113 1676 consensus_queue.cc:237] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.785005 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:09:35.786641 1568 leader_election.cc:341] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:35.788244 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:35.788833 1431 tablet_service.cc:1467] Processing CreateTablet for tablet 2355fe5dfc824ec78c044695be9d32de (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
W20260808 01:09:35.789690 1567 leader_election.cc:341] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:35.790084 1431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2355fe5dfc824ec78c044695be9d32de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.790109 1567 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:35.791417 1649 raft_consensus.cc:2726] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:35.792039 1648 tablet_bootstrap.cc:492] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:35.792228 1514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f1af2e7c504c919e34f12663452f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.795514 1511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e2d40d6b2943d883579bc4571ed4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.780839 1436 rpcz_store.cc:276] 0808 01:09:34.735321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.354287 (+618966us) service_pool.cc:227] Handling call
0808 01:09:35.411625 (+ 57338us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.411730 (+ 105us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.531006 (+119276us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.773226 (+242220us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.773731 (+ 505us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:09:35.801903 1646 tablet_bootstrap.cc:492] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:35.804065 1508 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58250 (request call id 43) took 1097 ms. Trace:
I20260808 01:09:35.767323 1435 tablet_service.cc:1467] Processing CreateTablet for tablet a27b15da83cb42d49d1b0a703b5955ea (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260808 01:09:35.804184 1646 ts_tablet_manager.cc:1397] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.156s user 0.031s sys 0.016s
I20260808 01:09:35.802448 1436 tablet_service.cc:1467] Processing CreateTablet for tablet a668c2efd2dd4f5b946b70904c143c17 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260808 01:09:35.817073 1677 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:35.833868 1677 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.834897 1646 raft_consensus.cc:348] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.804852 1512 tablet_service.cc:1467] Processing CreateTablet for tablet b3583c46ad2f4aa8875c06ef46cde18a (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260808 01:09:35.825295 1506 tablet_service.cc:1467] Processing CreateTablet for tablet a27b15da83cb42d49d1b0a703b5955ea (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260808 01:09:35.828406 1435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a27b15da83cb42d49d1b0a703b5955ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.804464 1508 rpcz_store.cc:276] 0808 01:09:34.714600 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.433798 (+719198us) service_pool.cc:227] Handling call
0808 01:09:35.477861 (+ 44063us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.477967 (+ 106us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.708951 (+230984us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.791200 (+ 82249us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.803998 (+ 12798us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:09:35.841399 1429 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55036 (request call id 46) took 1102 ms. Trace:
I20260808 01:09:35.841804 1429 rpcz_store.cc:276] 0808 01:09:34.738801 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.367579 (+628778us) service_pool.cc:227] Handling call
0808 01:09:35.424508 (+ 56929us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.424611 (+ 103us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.769294 (+344683us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.818757 (+ 49463us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.841333 (+ 22576us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:09:35.842562 1508 tablet_service.cc:1467] Processing CreateTablet for tablet a3ce33a519294ae1bce66c8aca8cf601 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260808 01:09:35.836244 1677 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:35.837216 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:35.844092 1447 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:35.845249 1490 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:35.846225 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:35.847491 1512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3583c46ad2f4aa8875c06ef46cde18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.848241 1436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a668c2efd2dd4f5b946b70904c143c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.851059 1648 tablet_bootstrap.cc:654] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:35.849145 1340 catalog_manager.cc:5495] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:35.851402 1677 raft_consensus.cc:2781] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:35.855295 1677 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:35.855687 1677 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:35.844576 1429 tablet_service.cc:1467] Processing CreateTablet for tablet 1ccc57605ae24bf4bacee4e4741bbe95 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260808 01:09:35.853348 1508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ce33a519294ae1bce66c8aca8cf601. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.839691 1646 raft_consensus.cc:374] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:35.857673 1646 raft_consensus.cc:729] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:35.858490 1646 consensus_queue.cc:260] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.860307 1432 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55036 (request call id 49) took 1006 ms. Trace:
I20260808 01:09:35.860689 1432 rpcz_store.cc:276] 0808 01:09:34.853638 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.500069 (+646431us) service_pool.cc:227] Handling call
0808 01:09:35.528285 (+ 28216us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.528553 (+ 268us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.801761 (+273208us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.859757 (+ 57996us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.860239 (+ 482us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:09:35.864174 1506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a27b15da83cb42d49d1b0a703b5955ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.862294 1603 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:35.860805 1646 ts_tablet_manager.cc:1428] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.029s user 0.004s sys 0.002s
I20260808 01:09:35.870040 1429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccc57605ae24bf4bacee4e4741bbe95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.870157 1646 tablet_bootstrap.cc:492] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:35.874116 1677 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.876042 1646 tablet_bootstrap.cc:654] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:35.877045 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:35.877631 1447 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:35.883618 1510 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58250 (request call id 46) took 1085 ms. Trace:
I20260808 01:09:35.884109 1510 rpcz_store.cc:276] 0808 01:09:34.797858 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.586476 (+788618us) service_pool.cc:227] Handling call
0808 01:09:35.601577 (+ 15101us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.601692 (+ 115us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.784837 (+183145us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.839066 (+ 54229us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.883550 (+ 44484us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30000}
I20260808 01:09:35.886008 1510 tablet_service.cc:1467] Processing CreateTablet for tablet 1ccc57605ae24bf4bacee4e4741bbe95 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260808 01:09:35.887272 1510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccc57605ae24bf4bacee4e4741bbe95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.885430 1507 tablet_service.cc:1467] Processing CreateTablet for tablet 2355fe5dfc824ec78c044695be9d32de (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260808 01:09:35.887574 1447 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:35.888269 1507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2355fe5dfc824ec78c044695be9d32de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.888826 1490 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:35.889750 1668 raft_consensus.cc:2781] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:35.890259 1668 raft_consensus.cc:686] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:35.891116 1668 consensus_queue.cc:237] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.893147 1509 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58250 (request call id 49) took 1037 ms. Trace:
I20260808 01:09:35.894234 1645 tablet_bootstrap.cc:492] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:35.897163 1645 ts_tablet_manager.cc:1397] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.320s user 0.026s sys 0.014s
I20260808 01:09:35.897976 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:35.898662 1603 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:35.903229 1515 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58250 (request call id 44) took 1175 ms. Trace:
I20260808 01:09:35.903630 1515 rpcz_store.cc:276] 0808 01:09:34.735809 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.509533 (+773724us) service_pool.cc:227] Handling call
0808 01:09:35.511529 (+ 1996us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.511623 (+ 94us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.824274 (+312651us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.902753 (+ 78479us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.903171 (+ 418us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:09:35.901018 1677 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:35.906061 1513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58250 (request call id 45) took 1145 ms. Trace:
I20260808 01:09:35.906574 1513 rpcz_store.cc:276] 0808 01:09:34.760294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.535067 (+774773us) service_pool.cc:227] Handling call
0808 01:09:35.544742 (+ 9675us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.544850 (+ 108us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.775441 (+230591us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.895978 (+120537us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.905989 (+ 10011us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:09:35.908847 1645 raft_consensus.cc:348] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.909762 1645 raft_consensus.cc:374] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:35.910091 1645 raft_consensus.cc:729] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:35.910893 1645 consensus_queue.cc:260] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.896049 1509 rpcz_store.cc:276] 0808 01:09:34.855926 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.678161 (+822235us) service_pool.cc:227] Handling call
0808 01:09:35.702420 (+ 24259us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.720423 (+ 18003us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.807745 (+ 87322us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.892631 (+ 84886us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.893088 (+ 457us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:09:35.921402 1511 tablet_service.cc:1467] Processing CreateTablet for tablet a668c2efd2dd4f5b946b70904c143c17 (DEFAULT_TABLE table=test-workload [id=90f6392c55104263af6d6d1c9f7080de]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260808 01:09:35.942813 1511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a668c2efd2dd4f5b946b70904c143c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:35.940259 1603 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:35.949796 1645 ts_tablet_manager.cc:1428] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.052s user 0.006s sys 0.002s
I20260808 01:09:35.951120 1645 tablet_bootstrap.cc:492] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:35.956983 1646 tablet_bootstrap.cc:492] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:35.962514 1646 ts_tablet_manager.cc:1397] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.093s user 0.005s sys 0.014s
I20260808 01:09:35.957069 1648 tablet_bootstrap.cc:492] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:35.961670 1512 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58250 (request call id 53) took 1008 ms. Trace:
I20260808 01:09:35.964880 1512 rpcz_store.cc:276] 0808 01:09:34.952830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.803939 (+851109us) service_pool.cc:227] Handling call
0808 01:09:35.847026 (+ 43087us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.847126 (+ 100us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.916113 (+ 68987us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.960967 (+ 44854us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.961604 (+ 637us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:09:35.967420 1341 catalog_manager.cc:5495] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:35.969592 1514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58250 (request call id 50) took 1059 ms. Trace:
I20260808 01:09:35.969842 1514 rpcz_store.cc:276] 0808 01:09:34.910685 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:35.702343 (+791658us) service_pool.cc:227] Handling call
0808 01:09:35.752538 (+ 50195us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0808 01:09:35.753424 (+ 886us) ts_tablet_manager.cc:676] Creating new metadata...
0808 01:09:35.909630 (+156206us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:35.969198 (+ 59568us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:35.969546 (+ 348us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:09:35.973517 1646 raft_consensus.cc:348] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.964684 1648 ts_tablet_manager.cc:1397] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.173s user 0.015s sys 0.010s
I20260808 01:09:35.974143 1646 raft_consensus.cc:374] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:35.975935 1646 raft_consensus.cc:729] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:35.977332 1648 raft_consensus.cc:348] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.981383 1645 tablet_bootstrap.cc:654] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:35.982769 1648 raft_consensus.cc:374] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:35.984243 1648 raft_consensus.cc:729] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:35.983799 1646 consensus_queue.cc:260] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.985069 1648 consensus_queue.cc:260] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:35.989041 1648 ts_tablet_manager.cc:1428] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20260808 01:09:35.990432 1648 tablet_bootstrap.cc:492] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:35.991464 1645 tablet_bootstrap.cc:492] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:35.992020 1645 ts_tablet_manager.cc:1397] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.005s
I20260808 01:09:35.992352 1646 ts_tablet_manager.cc:1428] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.029s user 0.004s sys 0.004s
I20260808 01:09:35.993568 1646 tablet_bootstrap.cc:492] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:36.000800 1648 tablet_bootstrap.cc:654] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.002911 1646 tablet_bootstrap.cc:654] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:35.995067 1645 raft_consensus.cc:348] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.003993 1645 raft_consensus.cc:374] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.004323 1645 raft_consensus.cc:729] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.005497 1645 consensus_queue.cc:260] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.007457 1645 ts_tablet_manager.cc:1428] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.015s user 0.001s sys 0.004s
I20260808 01:09:36.008981 1645 tablet_bootstrap.cc:492] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:36.012110 1654 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.012729 1654 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.015304 1654 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.019754 1648 tablet_bootstrap.cc:492] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:36.021726 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:36.024029 1648 ts_tablet_manager.cc:1397] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.001s
W20260808 01:09:36.025349 1410 leader_election.cc:341] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.027693 1648 raft_consensus.cc:348] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.028767 1648 raft_consensus.cc:374] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.029132 1648 raft_consensus.cc:729] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.023067 1645 tablet_bootstrap.cc:654] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.022902 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:36.031663 1603 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:36.033442 1410 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:36.033855 1648 consensus_queue.cc:260] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.035108 1654 raft_consensus.cc:2781] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:36.036002 1654 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:36.036566 1654 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.037355 1648 ts_tablet_manager.cc:1428] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260808 01:09:36.038678 1648 tablet_bootstrap.cc:492] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:36.053256 1645 tablet_bootstrap.cc:492] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:36.053944 1645 ts_tablet_manager.cc:1397] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.005s
I20260808 01:09:36.057509 1645 raft_consensus.cc:348] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.058568 1645 raft_consensus.cc:374] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.058935 1645 raft_consensus.cc:729] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.060571 1676 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.061165 1676 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.065133 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:36.066483 1648 tablet_bootstrap.cc:654] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:36.067433 1568 leader_election.cc:341] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.068950 1676 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:36.060149 1645 consensus_queue.cc:260] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.064121 1654 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.071488 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:36.074239 1654 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.075872 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:09:36.076736 1567 leader_election.cc:341] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.076979 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:36.077584 1603 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.077345 1567 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:36.079003 1676 raft_consensus.cc:2726] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:36.083745 1646 tablet_bootstrap.cc:492] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
W20260808 01:09:36.084169 1410 leader_election.cc:341] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.084332 1646 ts_tablet_manager.cc:1397] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.091s user 0.037s sys 0.046s
I20260808 01:09:36.085871 1648 tablet_bootstrap.cc:492] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:36.086341 1648 ts_tablet_manager.cc:1397] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.048s user 0.012s sys 0.000s
I20260808 01:09:36.087237 1646 raft_consensus.cc:348] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.088032 1646 raft_consensus.cc:374] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.088320 1646 raft_consensus.cc:729] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.089020 1648 raft_consensus.cc:348] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.089234 1646 consensus_queue.cc:260] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.089227 1603 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:36.092190 1646 ts_tablet_manager.cc:1428] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 01:09:36.089787 1648 raft_consensus.cc:374] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.092693 1648 raft_consensus.cc:729] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.093428 1646 tablet_bootstrap.cc:492] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:36.093681 1648 consensus_queue.cc:260] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.094259 1668 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.094731 1668 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.095461 1648 ts_tablet_manager.cc:1428] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260808 01:09:36.096446 1648 tablet_bootstrap.cc:492] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:36.093691 1677 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.097186 1677 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.099107 1677 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.099888 1646 tablet_bootstrap.cc:654] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.103353 1648 tablet_bootstrap.cc:654] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.104727 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:36.105127 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:09:36.106814 1489 leader_election.cc:341] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:09:36.107662 1489 leader_election.cc:341] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.096726 1668 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.112656 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:36.113131 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:36.113397 1447 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:36.114471 1490 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:36.115505 1648 tablet_bootstrap.cc:492] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:36.115622 1668 raft_consensus.cc:2781] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:36.116135 1668 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:36.116281 1648 ts_tablet_manager.cc:1397] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.003s
I20260808 01:09:36.116511 1668 raft_consensus.cc:3037] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.120008 1648 raft_consensus.cc:348] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.120961 1648 raft_consensus.cc:374] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.121296 1648 raft_consensus.cc:729] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.122099 1654 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.122725 1654 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.112668 1646 tablet_bootstrap.cc:492] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:36.125245 1646 ts_tablet_manager.cc:1397] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.008s
I20260808 01:09:36.125345 1654 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:09:36.125705 1490 leader_election.cc:341] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.126185 1490 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:36.126327 1648 consensus_queue.cc:260] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.127300 1677 raft_consensus.cc:2726] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:36.128301 1410 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:36.129168 1646 raft_consensus.cc:348] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.130441 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:36.131543 1646 raft_consensus.cc:374] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.131697 1524 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:36.132085 1645 ts_tablet_manager.cc:1428] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.078s user 0.019s sys 0.025s
I20260808 01:09:36.131974 1646 raft_consensus.cc:729] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.133265 1645 tablet_bootstrap.cc:492] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:36.132984 1646 consensus_queue.cc:260] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.134661 1648 ts_tablet_manager.cc:1428] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.018s user 0.005s sys 0.002s
I20260808 01:09:36.135571 1648 tablet_bootstrap.cc:492] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:36.139639 1645 tablet_bootstrap.cc:654] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.139746 1646 ts_tablet_manager.cc:1428] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260808 01:09:36.140971 1646 tablet_bootstrap.cc:492] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:36.143428 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:36.144109 1603 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:36.145581 1648 tablet_bootstrap.cc:654] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.147152 1410 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:36.148036 1654 raft_consensus.cc:2781] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:36.148414 1654 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:36.148743 1654 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.131062 1697 raft_consensus.cc:2781] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:36.153271 1646 tablet_bootstrap.cc:654] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.158881 1677 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.159296 1654 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.159780 1677 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.163105 1677 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.165081 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:09:36.166852 1489 leader_election.cc:341] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.167385 1668 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.163767 1654 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.171190 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:09:36.172886 1489 leader_election.cc:341] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.173547 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:36.174309 1524 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.178148 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:36.178790 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:36.178937 1448 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:36.179378 1447 raft_consensus.cc:3037] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.180166 1645 tablet_bootstrap.cc:492] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:36.180258 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:36.180718 1645 ts_tablet_manager.cc:1397] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.048s user 0.014s sys 0.023s
I20260808 01:09:36.181036 1603 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.183597 1645 raft_consensus.cc:348] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.184320 1645 raft_consensus.cc:374] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.184600 1645 raft_consensus.cc:729] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.185329 1645 consensus_queue.cc:260] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.187332 1645 ts_tablet_manager.cc:1428] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 01:09:36.187412 1447 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:36.188448 1645 tablet_bootstrap.cc:492] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:36.188675 1490 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:36.189841 1677 raft_consensus.cc:2781] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:36.190248 1677 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:36.190696 1677 raft_consensus.cc:3037] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.193521 1697 raft_consensus.cc:686] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:36.194448 1697 consensus_queue.cc:237] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.195384 1645 tablet_bootstrap.cc:654] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.197970 1646 tablet_bootstrap.cc:492] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:36.198097 1677 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.200317 1646 ts_tablet_manager.cc:1397] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.007s
I20260808 01:09:36.201160 1677 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.198769 1490 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:36.202349 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:36.199163 1648 tablet_bootstrap.cc:492] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:36.202817 1677 raft_consensus.cc:2781] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:36.202891 1646 raft_consensus.cc:348] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.192694 1524 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:36.203438 1677 raft_consensus.cc:686] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
W20260808 01:09:36.203665 1489 leader_election.cc:341] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.203104 1648 ts_tablet_manager.cc:1397] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.068s user 0.010s sys 0.001s
I20260808 01:09:36.203679 1646 raft_consensus.cc:374] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.205214 1646 raft_consensus.cc:729] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.204700 1677 consensus_queue.cc:237] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.199613 1668 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.206750 1645 tablet_bootstrap.cc:492] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:36.207264 1648 raft_consensus.cc:348] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.215081 1645 ts_tablet_manager.cc:1397] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.003s
I20260808 01:09:36.215106 1646 consensus_queue.cc:260] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.215322 1648 raft_consensus.cc:374] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.216229 1648 raft_consensus.cc:729] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.202811 1603 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:36.216950 1648 consensus_queue.cc:260] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.218492 1645 raft_consensus.cc:348] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.219300 1645 raft_consensus.cc:374] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.207690 1410 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:36.219635 1645 raft_consensus.cc:729] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.220670 1697 raft_consensus.cc:2781] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:36.220409 1645 consensus_queue.cc:260] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.221132 1697 raft_consensus.cc:686] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:36.221956 1697 consensus_queue.cc:237] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.223014 1645 ts_tablet_manager.cc:1428] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 01:09:36.223552 1676 consensus_queue.cc:1035] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:36.223762 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:36.223984 1645 tablet_bootstrap.cc:492] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:36.224418 1447 raft_consensus.cc:3037] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.225035 1646 ts_tablet_manager.cc:1428] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.024s user 0.005s sys 0.009s
I20260808 01:09:36.232385 1447 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:36.233448 1645 tablet_bootstrap.cc:654] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.233850 1490 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:36.234704 1668 raft_consensus.cc:2781] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:36.235455 1668 raft_consensus.cc:686] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:36.236424 1668 consensus_queue.cc:237] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.236457 1340 catalog_manager.cc:5495] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:36.219005 1648 ts_tablet_manager.cc:1428] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.014s user 0.005s sys 0.003s
I20260808 01:09:36.239341 1648 tablet_bootstrap.cc:492] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:36.245369 1645 tablet_bootstrap.cc:492] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:36.245832 1645 ts_tablet_manager.cc:1397] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.003s
I20260808 01:09:36.246933 1648 tablet_bootstrap.cc:654] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.245924 1342 catalog_manager.cc:5495] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:36.248564 1645 raft_consensus.cc:348] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.249608 1645 raft_consensus.cc:374] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.249893 1645 raft_consensus.cc:729] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.250725 1645 consensus_queue.cc:260] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.252876 1645 ts_tablet_manager.cc:1428] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 01:09:36.253775 1645 tablet_bootstrap.cc:492] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:36.261435 1645 tablet_bootstrap.cc:654] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.263675 1648 tablet_bootstrap.cc:492] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:36.264259 1648 ts_tablet_manager.cc:1397] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.002s
I20260808 01:09:36.266786 1648 raft_consensus.cc:348] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.267491 1648 raft_consensus.cc:374] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.268953 1646 tablet_bootstrap.cc:492] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:36.269279 1648 raft_consensus.cc:729] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.284693 1648 consensus_queue.cc:260] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.287679 1648 ts_tablet_manager.cc:1428] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.023s user 0.004s sys 0.000s
I20260808 01:09:36.289001 1648 tablet_bootstrap.cc:492] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:36.300858 1648 tablet_bootstrap.cc:654] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.307452 1646 tablet_bootstrap.cc:654] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.316195 1648 tablet_bootstrap.cc:492] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:36.316670 1648 ts_tablet_manager.cc:1397] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.000s
I20260808 01:09:36.338882 1697 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.342732 1648 raft_consensus.cc:348] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.343746 1648 raft_consensus.cc:374] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.344295 1648 raft_consensus.cc:729] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.345417 1648 consensus_queue.cc:260] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.348276 1648 ts_tablet_manager.cc:1428] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.031s user 0.004s sys 0.004s
I20260808 01:09:36.349484 1648 tablet_bootstrap.cc:492] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:36.351161 1340 catalog_manager.cc:5495] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:36.354501 1677 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.355187 1677 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.360587 1677 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.339627 1697 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.364102 1697 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.365619 1447 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.369849 1645 tablet_bootstrap.cc:492] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:36.370193 1648 tablet_bootstrap.cc:654] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.370380 1645 ts_tablet_manager.cc:1397] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.117s user 0.028s sys 0.022s
I20260808 01:09:36.370049 1341 catalog_manager.cc:5495] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:36.373870 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:36.375922 1448 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:36.365906 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:36.375120 1645 raft_consensus.cc:348] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.376802 1645 raft_consensus.cc:374] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.376998 1645 raft_consensus.cc:729] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.377429 1645 consensus_queue.cc:260] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.378904 1603 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:36.380125 1645 ts_tablet_manager.cc:1428] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20260808 01:09:36.380259 1489 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:36.384594 1645 tablet_bootstrap.cc:492] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:36.388000 1668 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:36.389264 1668 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:36.389281 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:36.389729 1668 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.390182 1646 tablet_bootstrap.cc:492] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:36.390645 1646 ts_tablet_manager.cc:1397] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.122s user 0.019s sys 0.018s
I20260808 01:09:36.394218 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:36.394975 1646 raft_consensus.cc:348] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.395663 1646 raft_consensus.cc:374] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.395989 1645 tablet_bootstrap.cc:654] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.395931 1646 raft_consensus.cc:729] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.397084 1646 consensus_queue.cc:260] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.401742 1648 tablet_bootstrap.cc:492] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:36.402299 1648 ts_tablet_manager.cc:1397] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.053s user 0.016s sys 0.007s
I20260808 01:09:36.402942 1410 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:36.405071 1648 raft_consensus.cc:348] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.405972 1648 raft_consensus.cc:374] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.406265 1648 raft_consensus.cc:729] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.406301 1649 consensus_queue.cc:1035] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:36.407124 1648 consensus_queue.cc:260] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.407781 1668 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.409776 1648 ts_tablet_manager.cc:1428] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260808 01:09:36.412755 1646 ts_tablet_manager.cc:1428] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.022s user 0.004s sys 0.004s
I20260808 01:09:36.413069 1648 tablet_bootstrap.cc:492] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:36.413657 1646 tablet_bootstrap.cc:492] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:36.425093 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:36.425369 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:36.426007 1447 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.426010 1603 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.430615 1668 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.440959 1645 tablet_bootstrap.cc:492] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:36.441485 1645 ts_tablet_manager.cc:1397] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.057s user 0.020s sys 0.025s
I20260808 01:09:36.443409 1697 raft_consensus.cc:2726] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:36.444705 1645 raft_consensus.cc:348] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.445740 1702 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.445482 1645 raft_consensus.cc:374] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.446242 1668 consensus_queue.cc:1035] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:36.447002 1645 raft_consensus.cc:729] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.446178 1702 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.447762 1645 consensus_queue.cc:260] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.449517 1702 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.450103 1645 ts_tablet_manager.cc:1428] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260808 01:09:36.451295 1645 tablet_bootstrap.cc:492] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:36.452732 1646 tablet_bootstrap.cc:654] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.460273 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:36.450924 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:09:36.462102 1410 leader_election.cc:341] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.462317 1525 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:36.455756 1668 consensus_queue.cc:1035] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260808 01:09:36.463604 1410 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:36.465667 1702 raft_consensus.cc:2781] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:36.466001 1702 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:36.466346 1702 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.466768 1447 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:36.468932 1603 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:36.470392 1489 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:36.476203 1645 tablet_bootstrap.cc:654] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.472487 1668 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:36.478035 1668 raft_consensus.cc:686] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:36.478946 1668 consensus_queue.cc:237] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.484350 1702 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.497831 1648 tablet_bootstrap.cc:654] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.504212 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:36.505169 1525 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.508993 1342 catalog_manager.cc:5495] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:36.511289 1702 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.512460 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:09:36.513852 1410 leader_election.cc:341] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.516924 1525 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:36.521034 1410 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:36.522640 1702 raft_consensus.cc:2781] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:36.523495 1702 raft_consensus.cc:686] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:36.524664 1702 consensus_queue.cc:237] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.533439 1646 tablet_bootstrap.cc:492] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:36.533936 1646 ts_tablet_manager.cc:1397] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.120s user 0.013s sys 0.005s
I20260808 01:09:36.542981 1677 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.544971 1677 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.548499 1648 tablet_bootstrap.cc:492] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:36.549075 1648 ts_tablet_manager.cc:1397] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.137s user 0.015s sys 0.009s
I20260808 01:09:36.549595 1646 raft_consensus.cc:348] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.550179 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:36.550402 1646 raft_consensus.cc:374] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.551414 1646 raft_consensus.cc:729] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
W20260808 01:09:36.552521 1489 leader_election.cc:341] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.550038 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:36.552301 1646 consensus_queue.cc:260] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.549583 1677 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.554348 1668 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:09:36.554311 1490 leader_election.cc:341] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.554868 1490 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:36.554868 1668 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.556201 1677 raft_consensus.cc:2726] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:36.557010 1668 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.557874 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:36.557826 1648 raft_consensus.cc:348] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.558614 1648 raft_consensus.cc:374] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.558611 1446 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:36.558718 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:36.559067 1648 raft_consensus.cc:729] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.559446 1604 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:36.560024 1490 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:36.560904 1668 raft_consensus.cc:2781] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:36.560843 1648 consensus_queue.cc:260] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.561405 1668 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:36.561853 1668 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.563627 1648 ts_tablet_manager.cc:1428] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.014s user 0.002s sys 0.005s
I20260808 01:09:36.564826 1646 ts_tablet_manager.cc:1428] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.031s user 0.007s sys 0.003s
I20260808 01:09:36.565730 1649 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.566309 1646 tablet_bootstrap.cc:492] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:36.566217 1649 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.568511 1649 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:36.569300 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:36.569715 1668 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:36.570768 1568 leader_election.cc:341] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.571761 1668 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.573083 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:36.573740 1446 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.576695 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:36.577322 1604 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.577565 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:09:36.579056 1567 leader_election.cc:341] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.579516 1567 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:09:36.580608 1390 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:09:36.580730 1649 raft_consensus.cc:2726] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:36.581899 1645 tablet_bootstrap.cc:492] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:36.582398 1645 ts_tablet_manager.cc:1397] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.131s user 0.028s sys 0.018s
I20260808 01:09:36.582341 1446 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:36.582608 1648 tablet_bootstrap.cc:492] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:36.583499 1490 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:36.584554 1668 raft_consensus.cc:2781] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:36.585086 1668 raft_consensus.cc:686] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:36.581851 1342 catalog_manager.cc:5495] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:36.586444 1649 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.586143 1668 consensus_queue.cc:237] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.586910 1649 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.589066 1649 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:36.590067 1604 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:36.591249 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:36.592199 1446 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:36.593667 1568 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:36.594528 1649 raft_consensus.cc:2781] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:36.594856 1649 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:36.595172 1649 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.595587 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:36.596524 1525 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:36.600010 1645 raft_consensus.cc:348] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.600747 1645 raft_consensus.cc:374] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.601018 1645 raft_consensus.cc:729] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.602331 1645 consensus_queue.cc:260] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.603845 1646 tablet_bootstrap.cc:654] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.604790 1645 ts_tablet_manager.cc:1428] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20260808 01:09:36.606276 1645 tablet_bootstrap.cc:492] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:36.608489 1649 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.611351 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:36.612025 1446 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.616645 1645 tablet_bootstrap.cc:654] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.618396 1677 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.618889 1677 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.620582 1446 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:36.621001 1677 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.621783 1568 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:36.622355 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:36.623402 1676 raft_consensus.cc:2781] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:36.625516 1648 tablet_bootstrap.cc:654] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:36.624481 1490 leader_election.cc:341] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.633539 1646 tablet_bootstrap.cc:492] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:36.633766 1645 tablet_bootstrap.cc:492] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:36.626183 1677 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.634003 1646 ts_tablet_manager.cc:1397] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.068s user 0.009s sys 0.001s
I20260808 01:09:36.634126 1645 ts_tablet_manager.cc:1397] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.004s
I20260808 01:09:36.634305 1677 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.626896 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:36.627776 1676 raft_consensus.cc:686] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:36.630259 1649 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:36.637385 1645 raft_consensus.cc:348] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.668331 1645 raft_consensus.cc:374] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.638628 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:36.668864 1645 raft_consensus.cc:729] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.637295 1646 raft_consensus.cc:348] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.669781 1646 raft_consensus.cc:374] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.670141 1646 raft_consensus.cc:729] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.669710 1645 consensus_queue.cc:260] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.647260 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:36.638054 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:36.671078 1525 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.669319 1604 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:36.676028 1702 consensus_queue.cc:1035] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:36.677141 1489 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:09:36.678251 1652 raft_consensus.cc:2781] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:36.678625 1652 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:36.669348 1603 raft_consensus.cc:2379] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 1: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:09:36.679051 1652 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.680241 1489 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:36.673146 1646 consensus_queue.cc:260] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.674986 1645 ts_tablet_manager.cc:1428] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.041s user 0.017s sys 0.021s
I20260808 01:09:36.638505 1677 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.685379 1677 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.674561 1676 consensus_queue.cc:237] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.684672 1645 tablet_bootstrap.cc:492] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:36.688376 1652 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.685887 1646 ts_tablet_manager.cc:1428] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.052s user 0.008s sys 0.000s
I20260808 01:09:36.691519 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:36.692308 1446 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.691890 1646 tablet_bootstrap.cc:492] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:36.694625 1652 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.699999 1645 tablet_bootstrap.cc:654] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.707341 1646 tablet_bootstrap.cc:654] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.715222 1645 tablet_bootstrap.cc:492] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:36.715742 1645 ts_tablet_manager.cc:1397] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.004s
I20260808 01:09:36.717689 1648 tablet_bootstrap.cc:492] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:36.721084 1648 ts_tablet_manager.cc:1397] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.139s user 0.012s sys 0.003s
I20260808 01:09:36.717067 1342 catalog_manager.cc:5495] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:36.721669 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:36.722558 1604 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.718286 1645 raft_consensus.cc:348] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.724690 1645 raft_consensus.cc:374] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.725001 1645 raft_consensus.cc:729] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.729957 1604 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:36.731185 1446 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:36.731465 1489 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:36.732642 1668 raft_consensus.cc:2781] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:36.733371 1668 raft_consensus.cc:686] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:36.734277 1668 consensus_queue.cc:237] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:36.735921 1489 consensus_peers.cc:487] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Couldn't send request to peer a4862d0938c84fc4bd840e4c17acae78. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:36.735347 1645 consensus_queue.cc:260] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.738663 1645 ts_tablet_manager.cc:1428] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260808 01:09:36.739002 1525 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:36.736744 1341 catalog_manager.cc:5495] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:36.739061 1648 raft_consensus.cc:348] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.741840 1646 tablet_bootstrap.cc:492] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:36.743196 1646 ts_tablet_manager.cc:1397] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.052s user 0.013s sys 0.002s
I20260808 01:09:36.744546 1645 tablet_bootstrap.cc:492] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:36.746035 1646 raft_consensus.cc:348] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.747008 1646 raft_consensus.cc:374] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.740128 1677 raft_consensus.cc:2726] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:36.742525 1648 raft_consensus.cc:374] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.747013 1702 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:36.748728 1648 raft_consensus.cc:729] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.748764 1604 raft_consensus.cc:3037] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.749591 1648 consensus_queue.cc:260] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.747407 1646 raft_consensus.cc:729] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.752835 1646 consensus_queue.cc:260] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.755580 1646 ts_tablet_manager.cc:1428] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.012s user 0.002s sys 0.002s
I20260808 01:09:36.756680 1646 tablet_bootstrap.cc:492] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:36.760272 1648 ts_tablet_manager.cc:1428] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.039s user 0.005s sys 0.002s
I20260808 01:09:36.761284 1648 tablet_bootstrap.cc:492] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:36.765655 1668 consensus_queue.cc:1035] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:36.766965 1648 tablet_bootstrap.cc:654] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.771629 1646 tablet_bootstrap.cc:654] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.776139 1702 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:36.776647 1697 consensus_queue.cc:1035] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:36.760382 1645 tablet_bootstrap.cc:654] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.785280 1342 catalog_manager.cc:5495] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:36.786516 1648 tablet_bootstrap.cc:492] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:36.787022 1648 ts_tablet_manager.cc:1397] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.004s
I20260808 01:09:36.794559 1645 tablet_bootstrap.cc:492] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:36.795039 1645 ts_tablet_manager.cc:1397] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.006s
I20260808 01:09:36.798122 1646 tablet_bootstrap.cc:492] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:36.798288 1668 consensus_queue.cc:1035] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:36.798727 1646 ts_tablet_manager.cc:1397] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.002s
I20260808 01:09:36.801591 1648 raft_consensus.cc:348] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.801843 1645 raft_consensus.cc:348] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.802636 1645 raft_consensus.cc:374] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.802346 1648 raft_consensus.cc:374] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.802901 1645 raft_consensus.cc:729] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.803112 1648 raft_consensus.cc:729] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.803860 1702 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.803737 1645 consensus_queue.cc:260] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.804270 1702 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.805078 1648 consensus_queue.cc:260] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.805755 1645 ts_tablet_manager.cc:1428] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260808 01:09:36.806161 1702 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.806607 1645 tablet_bootstrap.cc:492] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:36.804654 1646 raft_consensus.cc:348] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.807484 1646 raft_consensus.cc:374] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.807749 1646 raft_consensus.cc:729] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.809314 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:36.810125 1648 ts_tablet_manager.cc:1428] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260808 01:09:36.811019 1648 tablet_bootstrap.cc:492] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:36.810876 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:36.813082 1645 tablet_bootstrap.cc:654] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.814205 1697 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.814716 1697 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.816104 1646 consensus_queue.cc:260] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.816798 1697 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.819733 1646 ts_tablet_manager.cc:1428] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20260808 01:09:36.820744 1646 tablet_bootstrap.cc:492] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:36.819921 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:36.821178 1602 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
W20260808 01:09:36.822118 1410 leader_election.cc:341] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:09:36.823416 1410 leader_election.cc:341] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.823912 1410 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:36.824709 1697 raft_consensus.cc:2726] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:36.825925 1410 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:36.826874 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:36.828492 1697 raft_consensus.cc:2781] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:36.828975 1697 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:36.829375 1697 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.830585 1648 tablet_bootstrap.cc:654] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.832362 1645 tablet_bootstrap.cc:492] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:36.833308 1645 ts_tablet_manager.cc:1397] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.003s
W20260808 01:09:36.836437 1410 leader_election.cc:341] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.836571 1645 raft_consensus.cc:348] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.844112 1668 consensus_queue.cc:1035] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260808 01:09:36.844041 1645 raft_consensus.cc:374] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.850262 1648 tablet_bootstrap.cc:492] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:36.844041 1697 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.850699 1648 ts_tablet_manager.cc:1397] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.002s
I20260808 01:09:36.849941 1645 raft_consensus.cc:729] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.839198 1646 tablet_bootstrap.cc:654] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.851962 1645 consensus_queue.cc:260] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.855270 1645 ts_tablet_manager.cc:1428] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260808 01:09:36.856261 1645 tablet_bootstrap.cc:492] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:36.859323 1702 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.859194 1648 raft_consensus.cc:348] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.859962 1648 raft_consensus.cc:374] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.859853 1702 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.853819 1697 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.861914 1702 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.865666 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:36.867901 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:36.868508 1604 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.860291 1648 raft_consensus.cc:729] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.872368 1648 consensus_queue.cc:260] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.873199 1524 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.876426 1604 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:36.877079 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:36.877054 1646 tablet_bootstrap.cc:492] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:36.878264 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:36.878746 1646 ts_tablet_manager.cc:1397] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.016s
I20260808 01:09:36.879904 1645 tablet_bootstrap.cc:654] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.881438 1646 raft_consensus.cc:348] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.882531 1524 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:36.882134 1646 raft_consensus.cc:374] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.883666 1646 raft_consensus.cc:729] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.886644 1736 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.887676 1648 ts_tablet_manager.cc:1428] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.037s user 0.006s sys 0.001s
I20260808 01:09:36.887312 1736 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.888901 1648 tablet_bootstrap.cc:492] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
W20260808 01:09:36.889508 1410 leader_election.cc:341] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.890964 1410 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:36.893702 1668 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.894297 1668 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.896574 1648 tablet_bootstrap.cc:654] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.898253 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:36.899089 1604 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:36.899962 1668 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.901258 1489 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:36.903214 1736 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.903970 1736 raft_consensus.cc:2781] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:36.904379 1736 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:36.904788 1736 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.906352 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:36.905364 1645 tablet_bootstrap.cc:492] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
W20260808 01:09:36.908008 1489 leader_election.cc:341] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.908358 1645 ts_tablet_manager.cc:1397] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.052s user 0.016s sys 0.000s
I20260808 01:09:36.909224 1648 tablet_bootstrap.cc:492] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:36.909770 1648 ts_tablet_manager.cc:1397] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.004s
I20260808 01:09:36.911355 1645 raft_consensus.cc:348] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.912302 1645 raft_consensus.cc:374] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.912070 1648 raft_consensus.cc:348] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.912576 1645 raft_consensus.cc:729] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.912885 1648 raft_consensus.cc:374] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.913369 1648 raft_consensus.cc:729] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.913261 1645 consensus_queue.cc:260] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.913995 1648 consensus_queue.cc:260] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.916841 1648 ts_tablet_manager.cc:1428] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260808 01:09:36.918030 1648 tablet_bootstrap.cc:492] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:36.926826 1759 raft_consensus.cc:2781] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:36.927455 1645 ts_tablet_manager.cc:1428] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.019s user 0.005s sys 0.012s
I20260808 01:09:36.928416 1645 tablet_bootstrap.cc:492] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:36.929345 1648 tablet_bootstrap.cc:654] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.885097 1646 consensus_queue.cc:260] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.927399 1759 raft_consensus.cc:686] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:36.934696 1645 tablet_bootstrap.cc:654] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.935297 1759 consensus_queue.cc:237] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.936789 1646 ts_tablet_manager.cc:1428] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.058s user 0.007s sys 0.000s
I20260808 01:09:36.937788 1646 tablet_bootstrap.cc:492] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:36.948364 1648 tablet_bootstrap.cc:492] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:36.949028 1648 ts_tablet_manager.cc:1397] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.002s
I20260808 01:09:36.954530 1736 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.954879 1648 raft_consensus.cc:348] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.956947 1648 raft_consensus.cc:374] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.957332 1648 raft_consensus.cc:729] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.958890 1648 consensus_queue.cc:260] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.961715 1648 ts_tablet_manager.cc:1428] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
W20260808 01:09:36.961738 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 70872.6us
I20260808 01:09:36.962112 1645 tablet_bootstrap.cc:492] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:36.962704 1645 ts_tablet_manager.cc:1397] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.035s user 0.016s sys 0.014s
I20260808 01:09:36.962769 1648 tablet_bootstrap.cc:492] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:36.965394 1645 raft_consensus.cc:348] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.966549 1645 raft_consensus.cc:374] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.966926 1645 raft_consensus.cc:729] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.967677 1645 consensus_queue.cc:260] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.969285 1648 tablet_bootstrap.cc:654] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.969964 1645 ts_tablet_manager.cc:1428] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 01:09:36.970968 1645 tablet_bootstrap.cc:492] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:36.974270 1646 tablet_bootstrap.cc:654] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:36.977344 1410 leader_election.cc:341] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:36.977952 1410 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:36.978689 1645 tablet_bootstrap.cc:654] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:36.979614 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:36.979909 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:36.980321 1604 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.980609 1446 raft_consensus.cc:2454] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:36.982270 1648 tablet_bootstrap.cc:492] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:36.982769 1648 ts_tablet_manager.cc:1397] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.004s
I20260808 01:09:36.985487 1648 raft_consensus.cc:348] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.986339 1648 raft_consensus.cc:374] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:36.986586 1648 raft_consensus.cc:729] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:36.986521 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:36.987202 1446 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:36.987376 1648 consensus_queue.cc:260] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.988147 1757 raft_consensus.cc:2726] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:36.987568 1604 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:36.987155 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:36.989918 1489 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:36.990105 1448 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.989916 1648 ts_tablet_manager.cc:1428] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 01:09:36.990751 1668 raft_consensus.cc:2781] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:36.991286 1490 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:36.991664 1648 tablet_bootstrap.cc:492] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:36.991369 1668 raft_consensus.cc:686] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:36.992658 1668 consensus_queue.cc:237] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.993709 1677 raft_consensus.cc:2781] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:36.994084 1677 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:36.994415 1677 raft_consensus.cc:3037] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:36.996027 1652 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:36.996393 1736 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:36.997205 1757 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:36.997722 1757 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:36.998219 1448 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:37.001077 1645 tablet_bootstrap.cc:492] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:37.001119 1648 tablet_bootstrap.cc:654] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.001806 1645 ts_tablet_manager.cc:1397] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.005s
I20260808 01:09:37.004762 1645 raft_consensus.cc:348] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.006003 1759 consensus_queue.cc:1035] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:37.005995 1645 raft_consensus.cc:374] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.010082 1645 raft_consensus.cc:729] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.010931 1645 consensus_queue.cc:260] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.004073 1736 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:37.001832 1677 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.014919 1645 ts_tablet_manager.cc:1428] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20260808 01:09:37.016526 1645 tablet_bootstrap.cc:492] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:37.017510 1648 tablet_bootstrap.cc:492] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:37.017957 1648 ts_tablet_manager.cc:1397] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.000s
I20260808 01:09:37.026033 1757 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.026122 1646 tablet_bootstrap.cc:492] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:37.027007 1646 ts_tablet_manager.cc:1397] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.089s user 0.011s sys 0.002s
I20260808 01:09:37.029089 1648 raft_consensus.cc:348] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.032886 1648 raft_consensus.cc:374] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.033329 1648 raft_consensus.cc:729] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.030556 1646 raft_consensus.cc:348] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.034143 1646 raft_consensus.cc:374] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.034552 1646 raft_consensus.cc:729] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.035511 1646 consensus_queue.cc:260] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.037896 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:37.038817 1646 ts_tablet_manager.cc:1428] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260808 01:09:37.039510 1648 consensus_queue.cc:260] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.042951 1646 tablet_bootstrap.cc:492] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:37.040881 1525 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:37.044971 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:37.050604 1759 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.051149 1736 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.051115 1759 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.051894 1736 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.053622 1759 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.054585 1736 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.050908 1410 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:37.056310 1759 raft_consensus.cc:2781] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260808 01:09:37.056653 1410 leader_election.cc:341] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.056766 1759 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:37.057292 1759 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.058982 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:37.058888 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:37.059675 1604 raft_consensus.cc:2454] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:37.063275 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:09:37.065815 1490 leader_election.cc:341] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.066596 1648 ts_tablet_manager.cc:1428] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.048s user 0.008s sys 0.000s
I20260808 01:09:37.019213 1341 catalog_manager.cc:5495] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:37.067438 1648 tablet_bootstrap.cc:492] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:37.076056 1410 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:37.076370 1759 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.080308 1757 raft_consensus.cc:2781] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:37.086314 1757 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:37.086668 1757 raft_consensus.cc:3037] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.087261 1759 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.097179 1648 tablet_bootstrap.cc:654] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:37.102335 1410 leader_election.cc:341] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.110795 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:37.115358 1645 tablet_bootstrap.cc:654] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.119392 1646 tablet_bootstrap.cc:654] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.124840 1736 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260808 01:09:37.126104 1489 leader_election.cc:341] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.128993 1489 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:37.129354 1604 raft_consensus.cc:3037] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.133911 1339 catalog_manager.cc:5495] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:37.139885 1770 raft_consensus.cc:2726] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:37.135651 1677 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.144258 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:37.144878 1525 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.148159 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:09:37.152058 1489 leader_election.cc:341] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.141741 1770 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260808 01:09:37.158214 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:37.171476 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:37.172443 1448 raft_consensus.cc:3037] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.175663 1525 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:37.181879 1448 raft_consensus.cc:2454] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:37.189116 1757 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.196404 1648 tablet_bootstrap.cc:492] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:37.212129 1648 ts_tablet_manager.cc:1397] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.145s user 0.024s sys 0.024s
I20260808 01:09:37.199605 1646 tablet_bootstrap.cc:492] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:37.214030 1646 ts_tablet_manager.cc:1397] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.171s user 0.023s sys 0.026s
I20260808 01:09:37.216727 1646 raft_consensus.cc:348] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.217473 1646 raft_consensus.cc:374] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.217751 1646 raft_consensus.cc:729] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
W20260808 01:09:37.200747 1410 leader_election.cc:341] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.219036 1490 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:37.220715 1410 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:37.209652 1677 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.222242 1702 raft_consensus.cc:2781] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:37.222693 1677 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.224918 1677 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.225324 1754 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.225528 1646 consensus_queue.cc:260] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.226029 1754 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.227937 1646 ts_tablet_manager.cc:1428] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260808 01:09:37.228813 1646 tablet_bootstrap.cc:492] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:37.231678 1754 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.233319 1757 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.236578 1646 tablet_bootstrap.cc:654] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.239589 1676 consensus_queue.cc:1035] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:37.245877 1601 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.252326 1645 tablet_bootstrap.cc:492] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:37.253603 1645 ts_tablet_manager.cc:1397] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.237s user 0.012s sys 0.002s
I20260808 01:09:37.257118 1668 raft_consensus.cc:2781] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:37.257490 1646 tablet_bootstrap.cc:492] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:37.257697 1668 raft_consensus.cc:686] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:37.257988 1646 ts_tablet_manager.cc:1397] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.014s
I20260808 01:09:37.258725 1668 consensus_queue.cc:237] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.260921 1648 raft_consensus.cc:348] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.261780 1648 raft_consensus.cc:374] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.261313 1646 raft_consensus.cc:348] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.262089 1648 raft_consensus.cc:729] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.262274 1646 raft_consensus.cc:374] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.262635 1646 raft_consensus.cc:729] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
W20260808 01:09:37.263945 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 47948.4us
I20260808 01:09:37.263657 1646 consensus_queue.cc:260] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.264673 1648 consensus_queue.cc:260] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.268779 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:37.272773 1648 ts_tablet_manager.cc:1428] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.055s user 0.006s sys 0.000s
I20260808 01:09:37.273900 1648 tablet_bootstrap.cc:492] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:37.278245 1646 ts_tablet_manager.cc:1428] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20260808 01:09:37.279269 1646 tablet_bootstrap.cc:492] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
W20260808 01:09:37.279094 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 60177us
W20260808 01:09:37.279568 1489 leader_election.cc:341] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.285209 1646 tablet_bootstrap.cc:654] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.286433 1734 consensus_queue.cc:1035] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:37.287159 1676 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.287634 1676 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.291527 1676 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:37.258518 1645 raft_consensus.cc:348] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.292128 1645 raft_consensus.cc:374] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.292433 1645 raft_consensus.cc:729] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.292475 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:37.293912 1645 consensus_queue.cc:260] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.295423 1702 raft_consensus.cc:686] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
W20260808 01:09:37.297724 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 71899.7us
I20260808 01:09:37.298450 1645 ts_tablet_manager.cc:1428] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.044s user 0.007s sys 0.001s
I20260808 01:09:37.297910 1702 consensus_queue.cc:237] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.299466 1645 tablet_bootstrap.cc:492] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:37.299470 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:37.303099 1524 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:37.305119 1646 tablet_bootstrap.cc:492] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:37.305564 1646 ts_tablet_manager.cc:1397] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.001s
I20260808 01:09:37.308210 1646 raft_consensus.cc:348] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.308465 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:37.308912 1646 raft_consensus.cc:374] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.309204 1646 raft_consensus.cc:729] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.309455 1648 tablet_bootstrap.cc:654] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.309932 1646 consensus_queue.cc:260] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.310266 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:37.310916 1601 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:37.312477 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:37.313041 1602 raft_consensus.cc:3037] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.314879 1646 ts_tablet_manager.cc:1428] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260808 01:09:37.315905 1646 tablet_bootstrap.cc:492] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:37.322304 1646 tablet_bootstrap.cc:654] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.322826 1645 tablet_bootstrap.cc:654] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.322989 1648 tablet_bootstrap.cc:492] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:37.323593 1648 ts_tablet_manager.cc:1397] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.050s user 0.014s sys 0.000s
I20260808 01:09:37.324450 1668 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.324998 1668 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.326637 1648 raft_consensus.cc:348] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.327226 1668 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.327486 1648 raft_consensus.cc:374] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.327920 1648 raft_consensus.cc:729] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.328689 1648 consensus_queue.cc:260] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.329110 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:09:37.330684 1489 leader_election.cc:341] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.332190 1648 ts_tablet_manager.cc:1428] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260808 01:09:37.333993 1648 tablet_bootstrap.cc:492] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:37.334501 1339 catalog_manager.cc:5495] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:37.331295 1602 raft_consensus.cc:2454] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:37.336251 1646 tablet_bootstrap.cc:492] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:37.336704 1646 ts_tablet_manager.cc:1397] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.004s
I20260808 01:09:37.340709 1646 raft_consensus.cc:348] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.341451 1646 raft_consensus.cc:374] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.341703 1646 raft_consensus.cc:729] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.342370 1646 consensus_queue.cc:260] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.345373 1646 ts_tablet_manager.cc:1428] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260808 01:09:37.346268 1646 tablet_bootstrap.cc:492] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:37.348984 1648 tablet_bootstrap.cc:654] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.352996 1646 tablet_bootstrap.cc:654] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.354108 1645 tablet_bootstrap.cc:492] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:37.354775 1645 ts_tablet_manager.cc:1397] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.056s user 0.012s sys 0.016s
I20260808 01:09:37.358009 1734 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.358165 1645 raft_consensus.cc:348] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.358556 1734 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.359143 1645 raft_consensus.cc:374] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.359488 1645 raft_consensus.cc:729] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.360997 1734 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:37.360958 1645 consensus_queue.cc:260] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.362669 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:37.363293 1645 ts_tablet_manager.cc:1428] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260808 01:09:37.363296 1447 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:37.364353 1645 tablet_bootstrap.cc:492] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:37.364677 1568 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:37.365553 1734 raft_consensus.cc:2781] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:37.365876 1734 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:37.366196 1734 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.372891 1648 tablet_bootstrap.cc:492] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:37.373445 1648 ts_tablet_manager.cc:1397] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.040s user 0.018s sys 0.002s
I20260808 01:09:37.374207 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:37.374871 1447 raft_consensus.cc:2454] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:37.377032 1734 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.380000 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:37.380599 1448 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.381337 1645 tablet_bootstrap.cc:654] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.384414 1668 consensus_queue.cc:1035] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:37.391511 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:37.393172 1648 raft_consensus.cc:348] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.394177 1648 raft_consensus.cc:374] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.394515 1648 raft_consensus.cc:729] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.394984 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:37.395990 1447 raft_consensus.cc:2454] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:37.396278 1448 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:37.398069 1568 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:37.399626 1734 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:37.400080 1649 raft_consensus.cc:2781] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:37.400784 1649 raft_consensus.cc:686] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:37.400629 1648 consensus_queue.cc:260] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.401918 1649 consensus_queue.cc:237] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.404197 1648 ts_tablet_manager.cc:1428] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.030s user 0.006s sys 0.002s
I20260808 01:09:37.405460 1648 tablet_bootstrap.cc:492] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:37.410532 1702 consensus_queue.cc:1035] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:37.412971 1410 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:37.413304 1648 tablet_bootstrap.cc:654] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.416810 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:37.417158 1410 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:37.418998 1649 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.419502 1649 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.421976 1649 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:37.423911 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:37.424819 1448 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:37.426548 1568 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:37.414397 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:37.416194 1645 tablet_bootstrap.cc:492] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:37.427677 1649 raft_consensus.cc:2781] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:37.428062 1645 ts_tablet_manager.cc:1397] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.064s user 0.025s sys 0.015s
I20260808 01:09:37.428062 1649 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:37.430758 1649 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.436689 1567 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:37.437762 1649 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:37.438732 1567 leader_election.cc:341] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.440006 1734 raft_consensus.cc:2726] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:37.442119 1649 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:37.443138 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:37.444011 1448 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.438726 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:09:37.452087 1567 leader_election.cc:341] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:09:37.453588 1567 leader_election.cc:341] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.454560 1759 raft_consensus.cc:2781] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:37.454689 1702 raft_consensus.cc:2781] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:37.457607 1448 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:37.459484 1568 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:37.463352 1676 raft_consensus.cc:2781] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:37.463896 1676 raft_consensus.cc:686] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:37.464828 1676 consensus_queue.cc:237] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.467278 1648 tablet_bootstrap.cc:492] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:37.468981 1702 raft_consensus.cc:686] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:37.469522 1778 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.469928 1702 consensus_queue.cc:237] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.470193 1778 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.454993 1759 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:37.472965 1759 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.473182 1645 raft_consensus.cc:348] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.474212 1645 raft_consensus.cc:374] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.474550 1645 raft_consensus.cc:729] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.475302 1645 consensus_queue.cc:260] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.477730 1778 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.478405 1645 ts_tablet_manager.cc:1428] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.049s user 0.007s sys 0.000s
I20260808 01:09:37.490569 1645 tablet_bootstrap.cc:492] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:37.481990 1490 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:37.454223 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:37.492224 1736 raft_consensus.cc:2781] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260808 01:09:37.492353 1410 leader_election.cc:341] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.492697 1736 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:37.493044 1736 raft_consensus.cc:3037] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.495003 1770 consensus_queue.cc:1035] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:37.499065 1490 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
W20260808 01:09:37.501586 1567 leader_election.cc:341] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.487098 1759 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.503024 1677 raft_consensus.cc:2781] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:37.503926 1677 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:37.504302 1677 raft_consensus.cc:3037] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.467764 1648 ts_tablet_manager.cc:1397] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.063s user 0.019s sys 0.014s
I20260808 01:09:37.505329 1759 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.509280 1645 tablet_bootstrap.cc:654] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.510684 1646 tablet_bootstrap.cc:492] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:37.511227 1646 ts_tablet_manager.cc:1397] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.165s user 0.053s sys 0.058s
I20260808 01:09:37.513863 1648 raft_consensus.cc:348] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.514576 1648 raft_consensus.cc:374] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.514936 1648 raft_consensus.cc:729] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.514357 1646 raft_consensus.cc:348] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.515797 1646 raft_consensus.cc:374] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.515681 1648 consensus_queue.cc:260] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.516237 1646 raft_consensus.cc:729] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.507505 1702 consensus_queue.cc:1035] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260808 01:09:37.517311 1646 consensus_queue.cc:260] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.518060 1648 ts_tablet_manager.cc:1428] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260808 01:09:37.519038 1648 tablet_bootstrap.cc:492] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:37.520200 1646 ts_tablet_manager.cc:1428] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.009s user 0.001s sys 0.006s
I20260808 01:09:37.520418 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:37.521306 1646 tablet_bootstrap.cc:492] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:37.524475 1648 tablet_bootstrap.cc:654] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.523308 1339 catalog_manager.cc:5495] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:37.520931 1736 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.529937 1736 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.530619 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:37.531747 1646 tablet_bootstrap.cc:654] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:37.534577 1489 leader_election.cc:341] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.534937 1645 tablet_bootstrap.cc:492] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:37.535403 1645 ts_tablet_manager.cc:1397] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.045s user 0.011s sys 0.002s
I20260808 01:09:37.536195 1648 tablet_bootstrap.cc:492] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:37.536617 1648 ts_tablet_manager.cc:1397] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20260808 01:09:37.538050 1645 raft_consensus.cc:348] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.538770 1645 raft_consensus.cc:374] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.539022 1645 raft_consensus.cc:729] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.539505 1648 raft_consensus.cc:348] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.539806 1645 consensus_queue.cc:260] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.540316 1648 raft_consensus.cc:374] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.540729 1648 raft_consensus.cc:729] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.542433 1645 ts_tablet_manager.cc:1428] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.007s user 0.000s sys 0.008s
I20260808 01:09:37.543404 1645 tablet_bootstrap.cc:492] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:37.548976 1645 tablet_bootstrap.cc:654] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.541431 1648 consensus_queue.cc:260] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.553318 1648 ts_tablet_manager.cc:1428] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260808 01:09:37.557301 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:37.557627 1645 tablet_bootstrap.cc:492] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:37.558799 1645 ts_tablet_manager.cc:1397] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
W20260808 01:09:37.560698 1410 leader_election.cc:341] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.560644 1759 consensus_queue.cc:1035] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:37.561343 1736 consensus_queue.cc:1035] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 01:09:37.562649 1410 leader_election.cc:341] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.563079 1410 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:37.564384 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:37.565012 1601 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.565953 1778 raft_consensus.cc:2726] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:37.568526 1646 tablet_bootstrap.cc:492] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:37.569169 1646 ts_tablet_manager.cc:1397] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.030s
I20260808 01:09:37.558373 1648 tablet_bootstrap.cc:492] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:37.524094 1677 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.572278 1646 raft_consensus.cc:348] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.573489 1646 raft_consensus.cc:374] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.573967 1646 raft_consensus.cc:729] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.574918 1677 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.575297 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:37.574925 1646 consensus_queue.cc:260] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.575958 1446 raft_consensus.cc:3037] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.576213 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:37.576454 1601 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:37.577546 1410 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:09:37.577764 1489 leader_election.cc:341] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.578375 1702 raft_consensus.cc:2781] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:37.578867 1702 raft_consensus.cc:686] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:37.579689 1702 consensus_queue.cc:237] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.581832 1645 raft_consensus.cc:348] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.582501 1645 raft_consensus.cc:374] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.582796 1645 raft_consensus.cc:729] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.584244 1677 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.584753 1677 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.586941 1677 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.583478 1645 consensus_queue.cc:260] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.588153 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:37.589128 1645 ts_tablet_manager.cc:1428] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.030s user 0.002s sys 0.003s
I20260808 01:09:37.588930 1445 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:37.593298 1490 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:37.593674 1646 ts_tablet_manager.cc:1428] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260808 01:09:37.594164 1677 raft_consensus.cc:2781] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:37.594655 1646 tablet_bootstrap.cc:492] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:37.594596 1677 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:37.595100 1677 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.589789 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:37.596467 1601 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:37.601373 1648 tablet_bootstrap.cc:654] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.602633 1778 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.603209 1778 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.606606 1770 consensus_queue.cc:1035] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:37.607374 1676 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.608021 1676 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.630592 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:37.631916 1525 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.632262 1645 tablet_bootstrap.cc:492] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:37.637192 1668 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.638559 1668 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.641413 1668 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.642624 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:37.643172 1646 tablet_bootstrap.cc:654] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.641125 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:09:37.645618 1489 leader_election.cc:341] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.651938 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:37.649806 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:37.656610 1445 raft_consensus.cc:2379] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 1: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
W20260808 01:09:37.660660 1490 leader_election.cc:341] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:37.661603 1490 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:37.664893 1448 raft_consensus.cc:3037] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.665760 1770 raft_consensus.cc:2726] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:37.665653 1339 catalog_manager.cc:5495] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:37.670626 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:37.671773 1523 raft_consensus.cc:2454] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:37.677749 1734 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.678849 1410 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:37.678457 1734 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.680279 1645 tablet_bootstrap.cc:654] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:37.680745 1759 raft_consensus.cc:2781] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:37.686431 1759 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:37.686931 1759 raft_consensus.cc:3037] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.687556 1734 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:37.690106 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:37.690953 1445 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:37.692569 1770 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.692692 1568 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:37.680991 1676 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:37.693243 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:37.693065 1770 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.699777 1734 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.708899 1649 raft_consensus.cc:2781] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:37.688997 1778 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.720091 1734 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.726939 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:37.737564 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:37.719482 1649 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:37.742820 1649 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:09:37.744935 1390 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:09:37.727969 1522 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:37.750110 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:37.751073 1601 raft_consensus.cc:2454] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:37.749193 1567 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:37.760162 1676 raft_consensus.cc:3037] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:09:37.774739 1806 log.cc:924] Time spent T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.095s user 0.000s sys 0.001s
I20260808 01:09:37.780655 1668 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.781383 1668 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.791755 1734 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:37.795646 1668 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.800467 1446 raft_consensus.cc:2454] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:37.805713 1770 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.810355 1677 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.814025 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:37.815809 1600 raft_consensus.cc:2379] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 1: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:09:37.807004 1525 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:37.820950 1677 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.820677 1342 catalog_manager.cc:5495] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:37.824991 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:09:37.827252 1567 consensus_peers.cc:487] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 -> Peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Couldn't send request to peer 6b09dbefd24b4d52b385c4dff5697dc5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:37.832042 1342 catalog_manager.cc:5495] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:37.834759 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:37.843331 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:37.844094 1444 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:37.845800 1759 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.852614 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:37.852622 1445 raft_consensus.cc:2379] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 1: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:09:37.855592 1646 tablet_bootstrap.cc:492] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:37.848428 1525 raft_consensus.cc:2454] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:37.856132 1646 ts_tablet_manager.cc:1397] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.262s user 0.046s sys 0.047s
I20260808 01:09:37.853711 1759 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:37.859645 1648 tablet_bootstrap.cc:492] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:37.860404 1646 raft_consensus.cc:348] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.863662 1648 ts_tablet_manager.cc:1397] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.305s user 0.008s sys 0.022s
I20260808 01:09:37.863723 1646 raft_consensus.cc:374] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.864480 1646 raft_consensus.cc:729] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.865322 1646 consensus_queue.cc:260] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.861500 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:37.861634 1645 tablet_bootstrap.cc:492] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:37.866487 1645 ts_tablet_manager.cc:1397] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.277s user 0.011s sys 0.018s
I20260808 01:09:37.869306 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:37.867074 1446 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.869963 1600 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.872860 1649 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.874027 1604 raft_consensus.cc:2379] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 1: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:09:37.880601 1676 raft_consensus.cc:2726] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:37.889417 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:37.890057 1525 raft_consensus.cc:3037] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.891510 1677 consensus_queue.cc:1035] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:37.892689 1490 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:37.893953 1677 raft_consensus.cc:2781] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:37.894178 1490 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:37.894546 1677 raft_consensus.cc:686] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:37.895166 1770 raft_consensus.cc:2781] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:37.895509 1770 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:37.895955 1770 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.891072 1648 raft_consensus.cc:348] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.896661 1648 raft_consensus.cc:374] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.897053 1648 raft_consensus.cc:729] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.897926 1648 consensus_queue.cc:260] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.895462 1677 consensus_queue.cc:237] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.891458 1448 raft_consensus.cc:2454] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:37.924566 1646 ts_tablet_manager.cc:1428] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.068s user 0.024s sys 0.023s
I20260808 01:09:37.902035 1489 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:37.927356 1646 tablet_bootstrap.cc:492] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
W20260808 01:09:37.916735 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 55624.9us
I20260808 01:09:37.935043 1525 raft_consensus.cc:2454] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:37.944933 1446 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:37.969009 1645 raft_consensus.cc:348] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.980665 1645 raft_consensus.cc:374] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:37.980957 1645 raft_consensus.cc:729] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:37.950482 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:37.959486 1649 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:37.965600 1814 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:37.983286 1645 consensus_queue.cc:260] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.991807 1645 ts_tablet_manager.cc:1428] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.125s user 0.005s sys 0.003s
I20260808 01:09:37.992974 1645 tablet_bootstrap.cc:492] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:37.987340 1448 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:37.988644 1817 consensus_queue.cc:1035] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
W20260808 01:09:37.987504 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 98904us
I20260808 01:09:37.997884 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:37.989146 1600 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:38.000712 1648 ts_tablet_manager.cc:1428] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.137s user 0.025s sys 0.023s
I20260808 01:09:38.001463 1648 tablet_bootstrap.cc:492] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:37.987943 1814 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:37.994787 1649 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260808 01:09:38.009358 1646 tablet_bootstrap.cc:654] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:38.001987 1604 raft_consensus.cc:3037] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:38.012861 1814 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:38.018517 1489 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:37.995606 1567 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:09:38.021142 1824 raft_consensus.cc:2781] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:38.024291 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:38.025816 1820 raft_consensus.cc:3037] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:38.048893 1490 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:38.058257 1447 raft_consensus.cc:2379] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 1: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:09:38.064947 1823 raft_consensus.cc:2781] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:38.049947 1410 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
W20260808 01:09:38.062233 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 68372.9us
I20260808 01:09:38.050774 1824 raft_consensus.cc:686] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:38.065760 1823 raft_consensus.cc:686] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:38.095477 1824 consensus_queue.cc:237] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.096692 1823 consensus_queue.cc:237] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.077697 1604 raft_consensus.cc:2454] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
W20260808 01:09:38.113318 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 72571.3us
I20260808 01:09:38.114416 1817 raft_consensus.cc:2741] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260808 01:09:38.114523 1826 raft_consensus.cc:2781] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:38.109457 1770 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.123267 1702 consensus_queue.cc:1035] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:38.131151 1770 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:38.138659 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:38.139940 1522 raft_consensus.cc:2379] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 1: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:09:38.144665 1648 tablet_bootstrap.cc:654] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:38.152850 1646 tablet_bootstrap.cc:492] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:38.155498 1826 raft_consensus.cc:686] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:38.155821 1646 ts_tablet_manager.cc:1397] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.229s user 0.032s sys 0.034s
I20260808 01:09:38.157734 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:38.161175 1448 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:38.161886 1525 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:38.162818 1568 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:38.164585 1817 raft_consensus.cc:2781] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:38.159695 1820 raft_consensus.cc:2726] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:09:38.166199 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 51583.5us
I20260808 01:09:38.170776 1567 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:38.172180 1817 raft_consensus.cc:686] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:38.181710 1816 raft_consensus.cc:3037] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:09:38.182253 1825 log.cc:924] Time spent T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260808 01:09:38.173825 1826 consensus_queue.cc:237] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.193332 1646 raft_consensus.cc:348] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.194196 1646 raft_consensus.cc:374] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:38.194471 1646 raft_consensus.cc:729] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:38.195104 1646 consensus_queue.cc:260] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.202656 1645 tablet_bootstrap.cc:654] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:38.205654 1817 consensus_queue.cc:237] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.195513 1831 consensus_queue.cc:1035] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260808 01:09:38.218488 1646 ts_tablet_manager.cc:1428] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.062s user 0.007s sys 0.003s
I20260808 01:09:38.219658 1820 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:38.222214 1820 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.224727 1820 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:38.221412 1770 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:38.226435 1770 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.206085 1832 consensus_queue.cc:1035] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260808 01:09:38.230876 1770 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:38.240237 1646 tablet_bootstrap.cc:492] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:38.255486 1820 consensus_queue.cc:1035] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:38.257318 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:38.259666 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:09:38.240532 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 44630.2us
W20260808 01:09:38.263824 1410 consensus_peers.cc:487] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Couldn't send request to peer a4862d0938c84fc4bd840e4c17acae78. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:38.263538 1734 consensus_queue.cc:1035] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260808 01:09:38.273907 1648 tablet_bootstrap.cc:492] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:38.274585 1648 ts_tablet_manager.cc:1397] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.273s user 0.009s sys 0.014s
I20260808 01:09:38.286118 1840 consensus_queue.cc:1035] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:38.288616 1645 tablet_bootstrap.cc:492] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:38.289336 1645 ts_tablet_manager.cc:1397] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.296s user 0.014s sys 0.015s
I20260808 01:09:38.289966 1342 catalog_manager.cc:5495] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:38.293318 1645 raft_consensus.cc:348] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.296522 1645 raft_consensus.cc:374] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:38.296905 1645 raft_consensus.cc:729] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:38.298792 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:38.297798 1645 consensus_queue.cc:260] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.313887 1645 ts_tablet_manager.cc:1428] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.024s user 0.008s sys 0.001s
I20260808 01:09:38.322085 1645 tablet_bootstrap.cc:492] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:38.326493 1524 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:38.331466 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:38.332428 1447 raft_consensus.cc:2379] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 1: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:09:38.334410 1646 tablet_bootstrap.cc:654] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:38.338601 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:38.340109 1648 raft_consensus.cc:348] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.340806 1648 raft_consensus.cc:374] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:38.341058 1648 raft_consensus.cc:729] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:38.341481 1648 consensus_queue.cc:260] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.334069 1816 raft_consensus.cc:2726] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:38.345340 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:38.347538 1645 tablet_bootstrap.cc:654] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:38.360909 1342 catalog_manager.cc:5495] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:38.369261 1648 ts_tablet_manager.cc:1428] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.094s user 0.006s sys 0.001s
I20260808 01:09:38.370190 1648 tablet_bootstrap.cc:492] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:38.370651 1342 catalog_manager.cc:5495] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:38.373143 1342 catalog_manager.cc:5495] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:38.381477 1646 tablet_bootstrap.cc:492] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:38.382009 1646 ts_tablet_manager.cc:1397] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.142s user 0.019s sys 0.027s
I20260808 01:09:38.385046 1646 raft_consensus.cc:348] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.386035 1646 raft_consensus.cc:374] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:38.386435 1646 raft_consensus.cc:729] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:38.387416 1646 consensus_queue.cc:260] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.403919 1646 ts_tablet_manager.cc:1428] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.022s user 0.008s sys 0.000s
I20260808 01:09:38.413378 1840 consensus_queue.cc:1035] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:38.417529 1648 tablet_bootstrap.cc:654] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:38.421656 1646 tablet_bootstrap.cc:492] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:38.423547 1490 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:38.428601 1820 raft_consensus.cc:2726] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:09:38.429867 1490 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:38.430579 1820 raft_consensus.cc:2726] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:38.431056 1490 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:38.437558 1342 catalog_manager.cc:5495] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:38.439761 1342 catalog_manager.cc:5495] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:38.448864 1702 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:38.449482 1702 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.451820 1702 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:38.455624 1668 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:38.455974 1820 raft_consensus.cc:2726] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:38.456821 1814 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:38.457695 1814 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.456198 1668 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.460407 1814 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:38.460736 1668 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:38.461313 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:38.462108 1447 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:38.463550 1568 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:38.466297 1814 raft_consensus.cc:2781] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:38.466840 1814 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:38.467384 1814 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:38.469123 1646 tablet_bootstrap.cc:654] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:38.470260 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:09:38.471716 1567 leader_election.cc:341] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:38.476279 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:09:38.478502 1489 leader_election.cc:341] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:38.483820 1840 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:38.484521 1840 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.486646 1840 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:38.491725 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:38.492082 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:38.492767 1523 raft_consensus.cc:2454] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:38.495499 1645 tablet_bootstrap.cc:492] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:38.496315 1645 ts_tablet_manager.cc:1397] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.179s user 0.014s sys 0.008s
I20260808 01:09:38.515007 1814 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.519434 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:38.519176 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:38.520433 1447 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:09:38.520709 1567 leader_election.cc:341] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:38.521752 1814 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:38.523653 1648 tablet_bootstrap.cc:492] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:38.524108 1648 ts_tablet_manager.cc:1397] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.154s user 0.009s sys 0.023s
I20260808 01:09:38.525717 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:38.528321 1645 raft_consensus.cc:348] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.529042 1645 raft_consensus.cc:374] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:38.529289 1645 raft_consensus.cc:729] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:38.529728 1646 tablet_bootstrap.cc:492] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:38.530164 1646 ts_tablet_manager.cc:1397] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.110s user 0.021s sys 0.015s
I20260808 01:09:38.529874 1645 consensus_queue.cc:260] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.530918 1648 raft_consensus.cc:348] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.531526 1648 raft_consensus.cc:374] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:38.531684 1648 raft_consensus.cc:729] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:38.532186 1648 consensus_queue.cc:260] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.535964 1646 raft_consensus.cc:348] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.536707 1646 raft_consensus.cc:374] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:38.536964 1646 raft_consensus.cc:729] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:38.537792 1646 consensus_queue.cc:260] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.537048 1410 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:38.539402 1648 ts_tablet_manager.cc:1428] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20260808 01:09:38.540401 1648 tablet_bootstrap.cc:492] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:38.526676 1522 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:38.544070 1447 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:38.546299 1646 ts_tablet_manager.cc:1428] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.016s user 0.006s sys 0.002s
I20260808 01:09:38.547073 1646 tablet_bootstrap.cc:492] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:38.548734 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:38.547763 1826 raft_consensus.cc:2781] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:38.549906 1826 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:38.549976 1604 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:38.550412 1826 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:09:38.554574 1489 consensus_peers.cc:487] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Couldn't send request to peer a4862d0938c84fc4bd840e4c17acae78. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:38.555461 1568 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:38.555603 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:09:38.557000 1410 leader_election.cc:341] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:09:38.595388 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 49815.8us
I20260808 01:09:38.584518 1814 raft_consensus.cc:2781] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:38.608152 1814 raft_consensus.cc:686] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
W20260808 01:09:38.608829 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 53471.2us
I20260808 01:09:38.609444 1814 consensus_queue.cc:237] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.612339 1826 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.614305 1826 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:38.617357 1648 tablet_bootstrap.cc:654] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:38.612972 1646 tablet_bootstrap.cc:654] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:38.630950 1410 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:38.633049 1645 ts_tablet_manager.cc:1428] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.136s user 0.012s sys 0.010s
I20260808 01:09:38.637857 1645 tablet_bootstrap.cc:492] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:38.637300 1826 raft_consensus.cc:2781] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:38.638525 1826 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:38.638968 1826 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:09:38.652791 1490 leader_election.cc:341] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:38.653338 1490 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:38.653930 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:38.655027 1645 tablet_bootstrap.cc:654] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:38.654857 1820 raft_consensus.cc:2726] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:38.655988 1524 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:38.660126 1648 tablet_bootstrap.cc:492] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:38.660590 1648 ts_tablet_manager.cc:1397] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.120s user 0.021s sys 0.010s
I20260808 01:09:38.662810 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:38.725238 1648 raft_consensus.cc:348] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.726118 1648 raft_consensus.cc:374] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:38.726431 1648 raft_consensus.cc:729] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:38.682250 1820 consensus_queue.cc:1035] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.045s
I20260808 01:09:38.679206 1524 raft_consensus.cc:2454] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:38.728257 1646 tablet_bootstrap.cc:492] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:38.728967 1646 ts_tablet_manager.cc:1397] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.182s user 0.011s sys 0.053s
I20260808 01:09:38.683677 1668 consensus_queue.cc:1035] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.046s
W20260808 01:09:38.713032 1410 leader_election.cc:341] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:38.665138 1339 catalog_manager.cc:5495] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:38.731966 1646 raft_consensus.cc:348] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.733086 1646 raft_consensus.cc:374] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:38.733424 1646 raft_consensus.cc:729] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:38.734102 1646 consensus_queue.cc:260] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.736017 1668 consensus_queue.cc:1035] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:38.736404 1646 ts_tablet_manager.cc:1428] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 01:09:38.737409 1646 tablet_bootstrap.cc:492] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:38.689975 1822 consensus_queue.cc:1035] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.062s
I20260808 01:09:38.727306 1648 consensus_queue.cc:260] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.746302 1826 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.760504 1648 ts_tablet_manager.cc:1428] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.100s user 0.008s sys 0.000s
I20260808 01:09:38.764286 1648 tablet_bootstrap.cc:492] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:38.757392 1770 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:38.769659 1770 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.776710 1759 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:38.777441 1816 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:38.777248 1759 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.777931 1816 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.779505 1759 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:38.781061 1816 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:38.782735 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:09:38.784565 1568 leader_election.cc:341] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:38.786237 1646 tablet_bootstrap.cc:654] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:38.793092 1822 consensus_queue.cc:1035] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:38.792073 1770 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:09:38.796828 1390 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:09:38.804116 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:38.805785 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:38.806494 1523 raft_consensus.cc:2454] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:38.808787 1567 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:09:38.813434 1410 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:38.813712 1816 raft_consensus.cc:2781] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:38.814091 1816 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:38.814450 1816 raft_consensus.cc:3037] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:38.814723 1826 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:38.815322 1759 raft_consensus.cc:2781] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:38.815905 1759 raft_consensus.cc:686] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:38.816263 1814 consensus_queue.cc:1035] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:38.817124 1759 consensus_queue.cc:237] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.820197 1814 consensus_queue.cc:1035] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:38.821857 1702 consensus_queue.cc:1035] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:38.839952 1645 tablet_bootstrap.cc:492] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:38.840509 1645 ts_tablet_manager.cc:1397] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.203s user 0.017s sys 0.027s
I20260808 01:09:38.850095 1814 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:38.857868 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:38.857079 1814 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.849992 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:38.863598 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:38.864410 1522 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:38.866340 1814 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:38.867576 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:38.868409 1601 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:38.870808 1816 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.871628 1645 raft_consensus.cc:348] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.876597 1645 raft_consensus.cc:374] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:38.876945 1645 raft_consensus.cc:729] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
W20260808 01:09:38.877511 1410 leader_election.cc:341] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:38.877612 1645 consensus_queue.cc:260] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.882201 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:38.883136 1447 raft_consensus.cc:2379] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 1: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:09:38.883539 1648 tablet_bootstrap.cc:654] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:38.884668 1865 log.cc:924] Time spent T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
W20260808 01:09:38.897760 1410 leader_election.cc:341] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:38.897867 1645 ts_tablet_manager.cc:1428] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.057s user 0.006s sys 0.003s
I20260808 01:09:38.898285 1410 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:38.899448 1645 tablet_bootstrap.cc:492] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:38.900516 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:38.913132 1646 tablet_bootstrap.cc:492] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:38.913651 1646 ts_tablet_manager.cc:1397] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.176s user 0.011s sys 0.006s
I20260808 01:09:38.916486 1646 raft_consensus.cc:348] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.917307 1646 raft_consensus.cc:374] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:38.917610 1646 raft_consensus.cc:729] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:38.918326 1646 consensus_queue.cc:260] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.920764 1646 ts_tablet_manager.cc:1428] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260808 01:09:38.922360 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:38.922726 1832 raft_consensus.cc:2726] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:38.925428 1646 tablet_bootstrap.cc:492] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:38.946084 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:09:38.949863 1568 leader_election.cc:341] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:09:38.950779 1568 leader_election.cc:341] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:38.951471 1816 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:38.952386 1490 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:38.986369 1601 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:38.988822 1410 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:38.990172 1778 raft_consensus.cc:2781] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:38.962604 1822 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:38.990800 1778 raft_consensus.cc:686] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:38.954757 1759 consensus_queue.cc:1035] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260808 01:09:38.998607 1778 consensus_queue.cc:237] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:38.999993 1522 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:39.049696 1648 tablet_bootstrap.cc:492] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:39.050334 1648 ts_tablet_manager.cc:1397] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.289s user 0.014s sys 0.018s
I20260808 01:09:39.053787 1646 tablet_bootstrap.cc:654] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:39.094339 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 62308.9us
I20260808 01:09:39.096453 1645 tablet_bootstrap.cc:654] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.104137 1814 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:39.105479 1814 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.112501 1814 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:09:39.115067 1567 leader_election.cc:341] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:39.094344 1833 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:39.120849 1833 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.123023 1567 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:39.130831 1833 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:39.132115 1814 raft_consensus.cc:2726] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:39.151190 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:39.152844 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:39.155062 1521 raft_consensus.cc:2454] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:39.166347 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:39.167429 1522 raft_consensus.cc:3037] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.166798 1648 raft_consensus.cc:348] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.169088 1648 raft_consensus.cc:374] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.169489 1648 raft_consensus.cc:729] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.170343 1648 consensus_queue.cc:260] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.172765 1648 ts_tablet_manager.cc:1428] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.122s user 0.003s sys 0.008s
I20260808 01:09:39.178535 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:39.179867 1521 raft_consensus.cc:2379] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 1: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:09:39.180181 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:39.179685 1822 raft_consensus.cc:2726] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:39.189208 1645 tablet_bootstrap.cc:492] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:39.190631 1645 ts_tablet_manager.cc:1397] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.292s user 0.036s sys 0.028s
I20260808 01:09:39.189940 1646 tablet_bootstrap.cc:492] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:39.191433 1646 ts_tablet_manager.cc:1397] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.266s user 0.015s sys 0.019s
I20260808 01:09:39.192055 1410 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:39.192984 1833 raft_consensus.cc:2781] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:39.193414 1833 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:39.193789 1833 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.194255 1567 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:39.195269 1816 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.191740 1875 consensus_queue.cc:1035] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:39.196602 1646 raft_consensus.cc:348] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.199110 1646 raft_consensus.cc:374] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.200042 1646 raft_consensus.cc:729] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.199728 1648 tablet_bootstrap.cc:492] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
W20260808 01:09:39.201040 1410 leader_election.cc:341] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:39.207737 1646 consensus_queue.cc:260] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.209900 1816 raft_consensus.cc:2726] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:39.212287 1645 raft_consensus.cc:348] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.218783 1645 raft_consensus.cc:374] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.219120 1645 raft_consensus.cc:729] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.216342 1646 ts_tablet_manager.cc:1428] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.024s user 0.009s sys 0.000s
I20260808 01:09:39.219923 1645 consensus_queue.cc:260] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.222721 1645 ts_tablet_manager.cc:1428] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.030s user 0.005s sys 0.001s
I20260808 01:09:39.222846 1646 tablet_bootstrap.cc:492] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:39.224494 1645 tablet_bootstrap.cc:492] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:39.231446 1833 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.236948 1648 tablet_bootstrap.cc:654] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.251266 1759 consensus_queue.cc:1035] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260808 01:09:39.252497 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:39.254930 1524 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.241868 1522 raft_consensus.cc:2454] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:39.254622 1816 consensus_queue.cc:1035] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260808 01:09:39.257217 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:39.247251 1833 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:09:39.262495 1410 leader_election.cc:341] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:39.270859 1833 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:39.272439 1341 catalog_manager.cc:5495] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:39.276363 1645 tablet_bootstrap.cc:654] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.276777 1833 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.273002 1567 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:09:39.280962 1341 catalog_manager.cc:5495] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:39.285712 1646 tablet_bootstrap.cc:654] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.295614 1814 raft_consensus.cc:2781] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:39.296748 1814 raft_consensus.cc:686] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:39.297951 1814 consensus_queue.cc:237] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.304083 1833 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:39.305537 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:39.306252 1522 raft_consensus.cc:2454] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:39.308004 1410 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:39.310508 1601 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.313349 1648 tablet_bootstrap.cc:492] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:39.314951 1648 ts_tablet_manager.cc:1397] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.117s user 0.018s sys 0.001s
I20260808 01:09:39.325196 1648 raft_consensus.cc:348] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.326135 1648 raft_consensus.cc:374] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.326418 1648 raft_consensus.cc:729] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.327099 1648 consensus_queue.cc:260] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.332996 1648 ts_tablet_manager.cc:1428] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20260808 01:09:39.334121 1648 tablet_bootstrap.cc:492] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:39.335551 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:39.336038 1832 raft_consensus.cc:2781] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:39.336419 1604 raft_consensus.cc:2454] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:39.336494 1832 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:39.339524 1832 raft_consensus.cc:3037] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.326649 1524 raft_consensus.cc:2454] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:39.347973 1410 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:39.348863 1826 raft_consensus.cc:2781] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:39.349316 1826 raft_consensus.cc:686] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:39.350238 1826 consensus_queue.cc:237] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.351332 1831 consensus_queue.cc:1035] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:39.362457 1645 tablet_bootstrap.cc:492] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:39.362964 1645 ts_tablet_manager.cc:1397] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.139s user 0.003s sys 0.017s
I20260808 01:09:39.370122 1645 raft_consensus.cc:348] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.371119 1645 raft_consensus.cc:374] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.371519 1645 raft_consensus.cc:729] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.372371 1645 consensus_queue.cc:260] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.373348 1648 tablet_bootstrap.cc:654] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.375571 1832 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.379092 1646 tablet_bootstrap.cc:492] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:39.379514 1646 ts_tablet_manager.cc:1397] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.157s user 0.010s sys 0.008s
I20260808 01:09:39.381743 1645 ts_tablet_manager.cc:1428] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260808 01:09:39.384426 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:39.385109 1523 raft_consensus.cc:3037] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.385636 1832 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:39.385389 1646 raft_consensus.cc:348] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.386233 1646 raft_consensus.cc:374] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.386564 1646 raft_consensus.cc:729] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.386584 1645 tablet_bootstrap.cc:492] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:39.390194 1646 consensus_queue.cc:260] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.391711 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:39.395159 1602 raft_consensus.cc:3037] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.401887 1646 ts_tablet_manager.cc:1428] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.022s user 0.006s sys 0.003s
I20260808 01:09:39.402101 1523 raft_consensus.cc:2454] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:39.403352 1410 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:39.404505 1832 raft_consensus.cc:2781] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:39.405102 1832 raft_consensus.cc:686] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:39.403209 1646 tablet_bootstrap.cc:492] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:39.405921 1832 consensus_queue.cc:237] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.413147 1645 tablet_bootstrap.cc:654] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.419903 1648 tablet_bootstrap.cc:492] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:39.420415 1648 ts_tablet_manager.cc:1397] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.087s user 0.009s sys 0.010s
I20260808 01:09:39.427950 1646 tablet_bootstrap.cc:654] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.433772 1602 raft_consensus.cc:2454] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:39.437991 1648 raft_consensus.cc:348] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.438730 1648 raft_consensus.cc:374] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.439014 1648 raft_consensus.cc:729] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.439639 1648 consensus_queue.cc:260] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.441476 1648 ts_tablet_manager.cc:1428] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20260808 01:09:39.442490 1648 tablet_bootstrap.cc:492] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:39.445998 1646 tablet_bootstrap.cc:492] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:39.446542 1646 ts_tablet_manager.cc:1397] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.044s user 0.012s sys 0.001s
I20260808 01:09:39.447619 1648 tablet_bootstrap.cc:654] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.449376 1646 raft_consensus.cc:348] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.450225 1646 raft_consensus.cc:374] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.453928 1341 catalog_manager.cc:5495] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:39.456167 1341 catalog_manager.cc:5495] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:39.450598 1646 raft_consensus.cc:729] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.461740 1646 consensus_queue.cc:260] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.465276 1646 ts_tablet_manager.cc:1428] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.018s user 0.004s sys 0.003s
I20260808 01:09:39.473701 1646 tablet_bootstrap.cc:492] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:39.475785 1645 tablet_bootstrap.cc:492] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:39.476341 1645 ts_tablet_manager.cc:1397] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.090s user 0.012s sys 0.019s
I20260808 01:09:39.479104 1645 raft_consensus.cc:348] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.479921 1645 raft_consensus.cc:374] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.480227 1645 raft_consensus.cc:729] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.480918 1645 consensus_queue.cc:260] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.481415 1648 tablet_bootstrap.cc:492] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:39.482009 1648 ts_tablet_manager.cc:1397] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.040s user 0.019s sys 0.013s
I20260808 01:09:39.483273 1646 tablet_bootstrap.cc:654] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.483711 1645 ts_tablet_manager.cc:1428] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 01:09:39.485018 1645 tablet_bootstrap.cc:492] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:39.485278 1648 raft_consensus.cc:348] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.486209 1648 raft_consensus.cc:374] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.486536 1648 raft_consensus.cc:729] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.487197 1648 consensus_queue.cc:260] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.482107 1339 catalog_manager.cc:5495] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:39.489406 1648 ts_tablet_manager.cc:1428] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260808 01:09:39.490437 1648 tablet_bootstrap.cc:492] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:39.491343 1645 tablet_bootstrap.cc:654] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.496901 1648 tablet_bootstrap.cc:654] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.511878 1646 tablet_bootstrap.cc:492] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:39.512567 1646 ts_tablet_manager.cc:1397] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.008s
I20260808 01:09:39.515091 1646 raft_consensus.cc:348] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.516101 1646 raft_consensus.cc:374] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.516518 1646 raft_consensus.cc:729] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.517204 1646 consensus_queue.cc:260] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.518450 1645 tablet_bootstrap.cc:492] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:39.518920 1645 ts_tablet_manager.cc:1397] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.020s
I20260808 01:09:39.519954 1646 ts_tablet_manager.cc:1428] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 01:09:39.521067 1646 tablet_bootstrap.cc:492] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:39.521459 1645 raft_consensus.cc:348] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.522156 1645 raft_consensus.cc:374] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.522410 1645 raft_consensus.cc:729] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.523022 1645 consensus_queue.cc:260] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.525051 1645 ts_tablet_manager.cc:1428] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 01:09:39.527630 1645 tablet_bootstrap.cc:492] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:39.528177 1668 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:39.528802 1668 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.531144 1668 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:39.531824 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:09:39.533664 1490 leader_election.cc:341] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:39.531368 1646 tablet_bootstrap.cc:654] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.535107 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:39.536055 1832 consensus_queue.cc:1035] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 01:09:39.537473 1489 leader_election.cc:341] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:39.537866 1489 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:39.538682 1668 raft_consensus.cc:2726] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:39.540489 1648 tablet_bootstrap.cc:492] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:39.546020 1648 ts_tablet_manager.cc:1397] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.056s user 0.017s sys 0.000s
I20260808 01:09:39.550192 1648 raft_consensus.cc:348] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.554539 1648 raft_consensus.cc:374] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.557765 1648 raft_consensus.cc:729] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.558553 1832 consensus_queue.cc:1035] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:39.558455 1648 consensus_queue.cc:260] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.560285 1668 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:39.560931 1648 ts_tablet_manager.cc:1428] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.014s user 0.005s sys 0.002s
I20260808 01:09:39.560791 1668 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.562064 1648 tablet_bootstrap.cc:492] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:39.564721 1645 tablet_bootstrap.cc:654] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.565361 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:09:39.568647 1489 leader_election.cc:341] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:39.569725 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:39.570395 1448 raft_consensus.cc:2454] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:39.570617 1646 tablet_bootstrap.cc:492] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:39.571135 1646 ts_tablet_manager.cc:1397] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.002s
I20260808 01:09:39.571933 1814 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:39.572440 1814 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.573979 1646 raft_consensus.cc:348] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.574635 1814 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:39.574765 1646 raft_consensus.cc:374] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.575076 1646 raft_consensus.cc:729] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.576880 1490 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:39.578238 1883 raft_consensus.cc:2781] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:39.578707 1883 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:39.579038 1883 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.579248 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:39.580265 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:39.580766 1648 tablet_bootstrap.cc:654] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:39.584226 1568 leader_election.cc:341] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:39.575764 1646 consensus_queue.cc:260] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.586117 1883 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:39.587801 1567 leader_election.cc:341] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:39.589578 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:39.589823 1567 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:39.590257 1833 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:39.590384 1448 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.590687 1875 raft_consensus.cc:2726] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:39.590806 1833 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.592943 1833 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:39.594152 1646 ts_tablet_manager.cc:1428] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.023s user 0.004s sys 0.004s
I20260808 01:09:39.595283 1646 tablet_bootstrap.cc:492] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:39.595697 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:39.596725 1883 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:39.596413 1524 raft_consensus.cc:2454] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:39.597998 1410 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:39.598680 1833 raft_consensus.cc:2781] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:39.599025 1833 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:39.600049 1833 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.604812 1448 raft_consensus.cc:2454] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:39.606068 1490 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:39.607077 1882 raft_consensus.cc:2781] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:39.563028 1668 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:39.607137 1833 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.609341 1646 tablet_bootstrap.cc:654] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.610266 1833 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:39.610895 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:39.614692 1648 tablet_bootstrap.cc:492] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:39.615209 1648 ts_tablet_manager.cc:1397] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.053s user 0.010s sys 0.003s
I20260808 01:09:39.616148 1882 raft_consensus.cc:686] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:39.617178 1882 consensus_queue.cc:237] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.618093 1648 raft_consensus.cc:348] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.618898 1648 raft_consensus.cc:374] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.619212 1648 raft_consensus.cc:729] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
W20260808 01:09:39.620604 1489 leader_election.cc:341] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:39.621205 1875 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:39.621699 1875 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.619995 1648 consensus_queue.cc:260] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.626284 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:39.626693 1875 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:39.626966 1522 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.628720 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:39.632679 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:39.634759 1522 raft_consensus.cc:2454] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:39.638113 1410 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:39.626289 1648 ts_tablet_manager.cc:1428] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260808 01:09:39.639055 1759 raft_consensus.cc:2781] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:39.639593 1759 raft_consensus.cc:686] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:39.640089 1648 tablet_bootstrap.cc:492] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:39.640514 1759 consensus_queue.cc:237] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:39.645323 1410 leader_election.cc:341] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:39.648280 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:39.648727 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:39.652218 1568 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:39.653380 1814 raft_consensus.cc:2726] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:39.657122 1646 tablet_bootstrap.cc:492] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:39.657594 1646 ts_tablet_manager.cc:1397] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.063s user 0.014s sys 0.002s
I20260808 01:09:39.660181 1646 raft_consensus.cc:348] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.660917 1646 raft_consensus.cc:374] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.661190 1646 raft_consensus.cc:729] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.663703 1648 tablet_bootstrap.cc:654] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.663604 1646 consensus_queue.cc:260] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.667109 1646 ts_tablet_manager.cc:1428] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260808 01:09:39.678169 1646 tablet_bootstrap.cc:492] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:39.689406 1339 catalog_manager.cc:5495] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
W20260808 01:09:39.692515 1410 leader_election.cc:341] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:39.693365 1646 tablet_bootstrap.cc:654] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.695144 1648 tablet_bootstrap.cc:492] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:39.699697 1648 ts_tablet_manager.cc:1397] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.060s user 0.013s sys 0.003s
I20260808 01:09:39.702272 1648 raft_consensus.cc:348] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.703016 1648 raft_consensus.cc:374] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.703311 1648 raft_consensus.cc:729] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.704035 1648 consensus_queue.cc:260] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.705051 1645 tablet_bootstrap.cc:492] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:39.705627 1645 ts_tablet_manager.cc:1397] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.178s user 0.051s sys 0.047s
I20260808 01:09:39.708060 1648 ts_tablet_manager.cc:1428] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260808 01:09:39.708498 1645 raft_consensus.cc:348] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.709169 1648 tablet_bootstrap.cc:492] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:39.709313 1645 raft_consensus.cc:374] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.709717 1645 raft_consensus.cc:729] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.710315 1645 consensus_queue.cc:260] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.712779 1645 ts_tablet_manager.cc:1428] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 01:09:39.713729 1645 tablet_bootstrap.cc:492] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:39.717247 1646 tablet_bootstrap.cc:492] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:39.717751 1646 ts_tablet_manager.cc:1397] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.000s
I20260808 01:09:39.720172 1645 tablet_bootstrap.cc:654] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.720631 1648 tablet_bootstrap.cc:654] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.720439 1646 raft_consensus.cc:348] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.721236 1646 raft_consensus.cc:374] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.721546 1646 raft_consensus.cc:729] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.722275 1646 consensus_queue.cc:260] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.725394 1341 catalog_manager.cc:5495] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:39.727010 1875 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:39.732190 1645 tablet_bootstrap.cc:492] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:39.732653 1645 ts_tablet_manager.cc:1397] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.000s
I20260808 01:09:39.727522 1875 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.733285 1646 ts_tablet_manager.cc:1428] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260808 01:09:39.734404 1646 tablet_bootstrap.cc:492] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:39.734777 1875 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:39.734973 1645 raft_consensus.cc:348] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.736429 1645 raft_consensus.cc:374] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.736671 1645 raft_consensus.cc:729] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.737283 1645 consensus_queue.cc:260] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.738763 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:39.739353 1645 ts_tablet_manager.cc:1428] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260808 01:09:39.739439 1521 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:39.740419 1645 tablet_bootstrap.cc:492] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:39.740974 1567 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:39.741333 1646 tablet_bootstrap.cc:654] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.743621 1816 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:39.744073 1816 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:39.744563 1816 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.748225 1646 tablet_bootstrap.cc:492] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:39.748569 1645 tablet_bootstrap.cc:654] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.748724 1646 ts_tablet_manager.cc:1397] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.003s
I20260808 01:09:39.748919 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:39.749662 1446 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:39.750840 1883 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:39.751336 1883 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.751690 1646 raft_consensus.cc:348] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.752509 1646 raft_consensus.cc:374] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.752802 1646 raft_consensus.cc:729] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.754410 1883 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:39.754843 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:09:39.757010 1489 leader_election.cc:341] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:39.753752 1646 consensus_queue.cc:260] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.759140 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:39.760607 1646 ts_tablet_manager.cc:1428] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260808 01:09:39.761862 1646 tablet_bootstrap.cc:492] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:39.763437 1602 raft_consensus.cc:3037] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.770756 1646 tablet_bootstrap.cc:654] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:39.771489 1490 leader_election.cc:341] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:39.772053 1490 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:39.773192 1778 consensus_queue.cc:1035] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:39.775193 1645 tablet_bootstrap.cc:492] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:39.775684 1645 ts_tablet_manager.cc:1397] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.011s
I20260808 01:09:39.775620 1668 raft_consensus.cc:2726] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:39.777125 1649 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:39.777720 1649 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.778743 1645 raft_consensus.cc:348] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.779472 1645 raft_consensus.cc:374] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.779745 1645 raft_consensus.cc:729] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.780691 1649 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:39.780426 1645 consensus_queue.cc:260] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.781098 1668 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:39.782776 1814 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:39.781561 1668 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.785828 1668 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:39.787621 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:39.788482 1601 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:39.789630 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:39.789980 1489 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:39.783288 1814 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:39.792131 1568 leader_election.cc:341] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:39.816437 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:39.817194 1446 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:39.818470 1814 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:39.822507 1648 tablet_bootstrap.cc:492] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:39.822979 1648 ts_tablet_manager.cc:1397] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.114s user 0.011s sys 0.001s
I20260808 01:09:39.825794 1816 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.827979 1816 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:39.782730 1645 ts_tablet_manager.cc:1428] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260808 01:09:39.831350 1645 tablet_bootstrap.cc:492] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:39.837586 1645 tablet_bootstrap.cc:654] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:39.838049 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 48199us
I20260808 01:09:39.839614 1822 raft_consensus.cc:2781] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:39.830061 1648 raft_consensus.cc:348] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.840137 1822 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:39.840271 1648 raft_consensus.cc:374] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.840626 1822 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.840627 1648 raft_consensus.cc:729] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.841585 1648 consensus_queue.cc:260] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.844054 1648 ts_tablet_manager.cc:1428] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20260808 01:09:39.845032 1648 tablet_bootstrap.cc:492] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:39.852052 1648 tablet_bootstrap.cc:654] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:39.861116 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:39.860294 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:39.862649 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:39.863606 1646 tablet_bootstrap.cc:492] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:39.868294 1523 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.868520 1646 ts_tablet_manager.cc:1397] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.107s user 0.043s sys 0.039s
I20260808 01:09:39.872179 1646 raft_consensus.cc:348] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.872879 1646 raft_consensus.cc:374] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.873132 1646 raft_consensus.cc:729] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.873709 1646 consensus_queue.cc:260] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.875108 1822 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.879227 1645 tablet_bootstrap.cc:492] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:39.879801 1645 ts_tablet_manager.cc:1397] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.049s user 0.013s sys 0.000s
I20260808 01:09:39.881251 1523 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:39.882138 1646 ts_tablet_manager.cc:1428] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20260808 01:09:39.883054 1646 tablet_bootstrap.cc:492] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:39.884769 1648 tablet_bootstrap.cc:492] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:39.888705 1648 ts_tablet_manager.cc:1397] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.007s
I20260808 01:09:39.889822 1646 tablet_bootstrap.cc:654] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:39.889731 1567 leader_election.cc:341] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:39.890309 1567 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:39.891404 1567 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:39.891755 1833 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:39.892225 1833 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.892712 1875 raft_consensus.cc:2726] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:39.893298 1875 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:39.893751 1875 raft_consensus.cc:686] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:39.894205 1833 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:39.894562 1875 consensus_queue.cc:237] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.895920 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:39.896703 1523 raft_consensus.cc:2379] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 1: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:09:39.890322 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:39.898730 1446 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.899042 1833 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:39.898981 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:39.900550 1833 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.901499 1410 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:39.902693 1778 raft_consensus.cc:3037] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.903074 1833 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:39.904953 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:09:39.906512 1410 leader_election.cc:341] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:39.890908 1645 raft_consensus.cc:348] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.908475 1645 raft_consensus.cc:374] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.908751 1645 raft_consensus.cc:729] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.909363 1645 consensus_queue.cc:260] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.896584 1822 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:39.909184 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:09:39.909740 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 115733us
W20260808 01:09:39.917452 1410 leader_election.cc:341] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:39.917846 1410 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:39.919443 1759 raft_consensus.cc:2726] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:39.919463 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:39.920480 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:39.921182 1445 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.921245 1442 raft_consensus.cc:2379] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 1: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:09:39.922499 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:39.922566 1648 raft_consensus.cc:348] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.927680 1648 raft_consensus.cc:374] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.928216 1648 raft_consensus.cc:729] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.929332 1648 consensus_queue.cc:260] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.923365 1601 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.926018 1649 consensus_queue.cc:1035] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260808 01:09:39.940994 1645 ts_tablet_manager.cc:1428] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.059s user 0.004s sys 0.002s
I20260808 01:09:39.937912 1446 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:39.946657 1645 tablet_bootstrap.cc:492] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:39.941447 1445 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:39.952772 1648 ts_tablet_manager.cc:1428] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.064s user 0.004s sys 0.003s
I20260808 01:09:39.947470 1778 raft_consensus.cc:2726] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:39.956547 1490 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:39.946617 1568 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:39.958534 1646 tablet_bootstrap.cc:492] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:39.963510 1649 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:39.964123 1646 ts_tablet_manager.cc:1397] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.081s user 0.015s sys 0.021s
I20260808 01:09:39.956597 1833 consensus_queue.cc:1035] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20260808 01:09:39.962281 1410 consensus_peers.cc:487] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Couldn't send request to peer a4862d0938c84fc4bd840e4c17acae78. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:39.964047 1822 raft_consensus.cc:2781] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:39.962316 1648 tablet_bootstrap.cc:492] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:39.972841 1822 raft_consensus.cc:686] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:39.973835 1822 consensus_queue.cc:237] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.986083 1646 raft_consensus.cc:348] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.987110 1646 raft_consensus.cc:374] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:39.987473 1646 raft_consensus.cc:729] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:39.990106 1882 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:40.001596 1882 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.002457 1883 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:40.003053 1883 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.990185 1646 consensus_queue.cc:260] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:39.997788 1601 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:40.026278 1645 tablet_bootstrap.cc:654] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:40.018779 1882 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:40.036536 1646 ts_tablet_manager.cc:1428] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.072s user 0.009s sys 0.000s
I20260808 01:09:40.043529 1646 tablet_bootstrap.cc:492] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:40.056136 1816 consensus_queue.cc:1035] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:40.058288 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:40.059006 1600 raft_consensus.cc:2454] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:40.062917 1649 raft_consensus.cc:2726] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:40.060073 1883 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:40.077900 1646 tablet_bootstrap.cc:654] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:40.084178 1648 tablet_bootstrap.cc:654] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:40.088209 1646 tablet_bootstrap.cc:492] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:40.088891 1646 ts_tablet_manager.cc:1397] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.046s user 0.015s sys 0.000s
I20260808 01:09:40.088212 1645 tablet_bootstrap.cc:492] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:40.089718 1645 ts_tablet_manager.cc:1397] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.146s user 0.017s sys 0.000s
I20260808 01:09:40.091477 1646 raft_consensus.cc:348] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.092258 1646 raft_consensus.cc:374] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:40.092530 1646 raft_consensus.cc:729] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:40.092129 1645 raft_consensus.cc:348] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.092864 1645 raft_consensus.cc:374] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:40.093139 1645 raft_consensus.cc:729] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:40.093830 1645 consensus_queue.cc:260] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.095543 1648 tablet_bootstrap.cc:492] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:40.096035 1648 ts_tablet_manager.cc:1397] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.134s user 0.005s sys 0.013s
I20260808 01:09:40.096096 1645 ts_tablet_manager.cc:1428] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 01:09:40.096374 1646 consensus_queue.cc:260] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.097234 1645 tablet_bootstrap.cc:492] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:40.099251 1646 ts_tablet_manager.cc:1428] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260808 01:09:40.098888 1648 raft_consensus.cc:348] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.099761 1648 raft_consensus.cc:374] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:40.100083 1648 raft_consensus.cc:729] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:40.100704 1648 consensus_queue.cc:260] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.102895 1646 tablet_bootstrap.cc:492] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:40.103585 1648 ts_tablet_manager.cc:1428] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260808 01:09:40.104488 1648 tablet_bootstrap.cc:492] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:40.104502 1645 tablet_bootstrap.cc:654] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:40.109257 1646 tablet_bootstrap.cc:654] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:40.110713 1648 tablet_bootstrap.cc:654] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:40.119230 1645 tablet_bootstrap.cc:492] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:40.119922 1645 ts_tablet_manager.cc:1397] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.001s
I20260808 01:09:40.019430 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:40.122853 1646 tablet_bootstrap.cc:492] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:40.123010 1448 raft_consensus.cc:2454] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:40.123333 1646 ts_tablet_manager.cc:1397] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.005s
I20260808 01:09:40.123271 1645 raft_consensus.cc:348] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.124497 1645 raft_consensus.cc:374] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:40.124948 1645 raft_consensus.cc:729] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:40.125826 1645 consensus_queue.cc:260] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.127005 1646 raft_consensus.cc:348] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.127673 1646 raft_consensus.cc:374] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:40.127957 1646 raft_consensus.cc:729] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:40.128631 1646 consensus_queue.cc:260] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.130816 1646 ts_tablet_manager.cc:1428] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260808 01:09:40.132112 1646 tablet_bootstrap.cc:492] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:40.136734 1645 ts_tablet_manager.cc:1428] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260808 01:09:40.136925 1648 tablet_bootstrap.cc:492] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:40.137351 1648 ts_tablet_manager.cc:1397] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.023s
I20260808 01:09:40.137686 1646 tablet_bootstrap.cc:654] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:40.137868 1645 tablet_bootstrap.cc:492] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:40.140875 1648 raft_consensus.cc:348] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.141646 1648 raft_consensus.cc:374] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:40.141934 1648 raft_consensus.cc:729] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:40.142545 1648 consensus_queue.cc:260] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.144690 1648 ts_tablet_manager.cc:1428] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 01:09:40.145627 1648 tablet_bootstrap.cc:492] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:40.147204 1645 tablet_bootstrap.cc:654] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:40.207322 1489 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:40.246240 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:40.254256 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:40.255007 1601 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:40.255554 1930 raft_consensus.cc:2781] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:40.255568 1926 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:40.256011 1930 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:40.256402 1930 raft_consensus.cc:3037] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:40.256323 1489 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:40.256208 1926 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.258661 1926 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:40.259991 1926 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:40.260418 1926 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.262234 1926 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:40.262302 1822 raft_consensus.cc:2781] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:40.262723 1822 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:40.263027 1822 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:40.270552 1930 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.275275 1930 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:40.277192 1822 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:40.282222 1490 leader_election.cc:341] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:40.282212 1822 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:40.288136 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:40.288836 1448 raft_consensus.cc:3037] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:40.297796 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:40.300532 1648 tablet_bootstrap.cc:654] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:40.301188 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:40.301800 1601 raft_consensus.cc:3037] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:40.302309 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:40.302915 1600 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:40.309188 1646 tablet_bootstrap.cc:492] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:40.295722 1448 raft_consensus.cc:2454] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:40.309731 1646 ts_tablet_manager.cc:1397] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.178s user 0.020s sys 0.011s
I20260808 01:09:40.312007 1668 consensus_queue.cc:1035] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:40.312803 1600 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:40.314755 1831 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:40.316815 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:40.317201 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:40.323024 1645 tablet_bootstrap.cc:492] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:40.323529 1645 ts_tablet_manager.cc:1397] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.186s user 0.013s sys 0.006s
I20260808 01:09:40.315239 1831 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.327124 1645 raft_consensus.cc:348] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.327957 1446 raft_consensus.cc:2454] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
W20260808 01:09:40.328696 1568 leader_election.cc:341] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:40.327589 1831 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:40.330981 1645 raft_consensus.cc:374] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:40.331280 1645 raft_consensus.cc:729] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:40.330796 1601 raft_consensus.cc:2454] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:40.333276 1645 consensus_queue.cc:260] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.337956 1602 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:09:40.339273 1390 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:09:40.340911 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:40.341646 1599 raft_consensus.cc:2454] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:40.341859 1646 raft_consensus.cc:348] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.342623 1646 raft_consensus.cc:374] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:40.342868 1645 ts_tablet_manager.cc:1428] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.019s user 0.009s sys 0.000s
I20260808 01:09:40.342902 1646 raft_consensus.cc:729] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:40.344044 1646 consensus_queue.cc:260] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:40.302426 1490 leader_election.cc:341] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:40.345384 1410 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:09:40.347523 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 45145.3us
I20260808 01:09:40.345201 1826 consensus_queue.cc:1035] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260808 01:09:40.349234 1759 raft_consensus.cc:2781] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:40.349661 1759 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:40.350005 1759 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:40.350780 1646 ts_tablet_manager.cc:1428] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.041s user 0.008s sys 0.000s
I20260808 01:09:40.363019 1645 tablet_bootstrap.cc:492] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:40.389269 1646 tablet_bootstrap.cc:492] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:40.411087 1568 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:40.429970 1814 raft_consensus.cc:2781] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:40.430421 1814 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:40.430824 1814 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:40.433022 1882 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:40.433701 1882 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.436240 1882 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:40.438520 1814 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.441890 1648 tablet_bootstrap.cc:492] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:40.442322 1648 ts_tablet_manager.cc:1397] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.297s user 0.014s sys 0.009s
I20260808 01:09:40.447749 1490 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:40.448949 1822 raft_consensus.cc:2781] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:40.449568 1822 raft_consensus.cc:686] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:40.450573 1822 consensus_queue.cc:237] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.451972 1943 consensus_queue.cc:1035] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:40.458778 1814 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:40.403635 1489 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:09:40.463543 1648 raft_consensus.cc:348] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.464931 1648 raft_consensus.cc:374] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:40.465240 1648 raft_consensus.cc:729] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:40.465786 1648 consensus_queue.cc:260] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.467636 1648 ts_tablet_manager.cc:1428] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20260808 01:09:40.467419 1342 catalog_manager.cc:5495] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:40.473255 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:40.477046 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:40.477749 1442 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:40.477948 1831 consensus_queue.cc:1035] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:40.478286 1814 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:40.478904 1814 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:40.480165 1410 leader_election.cc:341] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:40.458778 1759 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.481292 1648 tablet_bootstrap.cc:492] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:40.487025 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:40.490316 1930 raft_consensus.cc:2781] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
W20260808 01:09:40.491734 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 177822us
I20260808 01:09:40.492128 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:40.494059 1522 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:40.497000 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:40.497674 1599 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:40.512167 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:40.512478 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:40.513334 1446 raft_consensus.cc:2379] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 1: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:09:40.513803 1930 raft_consensus.cc:686] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:40.514676 1930 consensus_queue.cc:237] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:40.526805 1567 leader_election.cc:341] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:40.527778 1875 consensus_queue.cc:1035] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:40.526402 1831 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:40.528986 1831 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.531945 1831 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:40.534735 1814 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:40.531808 1759 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:40.532691 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:09:40.541415 1410 leader_election.cc:341] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:40.542058 1668 consensus_queue.cc:1035] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:40.542605 1490 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:40.543555 1668 raft_consensus.cc:2781] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:40.543861 1668 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:40.544142 1668 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:40.546784 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:40.545910 1567 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
W20260808 01:09:40.552618 1410 leader_election.cc:341] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:40.552675 1875 consensus_queue.cc:1035] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:40.553483 1816 raft_consensus.cc:2781] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:40.556306 1816 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:40.556639 1816 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:40.559309 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:40.561028 1649 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:40.554522 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:40.561638 1649 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.583334 1668 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.585994 1831 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:40.586396 1668 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:40.586742 1831 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:40.571715 1568 leader_election.cc:341] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:40.588099 1646 tablet_bootstrap.cc:654] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:40.592514 1645 tablet_bootstrap.cc:654] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:40.590584 1649 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:40.589387 1831 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:40.571913 1522 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:40.590544 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:40.602332 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:40.608553 1410 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:40.610041 1831 raft_consensus.cc:2781] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:40.610441 1831 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:09:40.612171 1410 leader_election.cc:341] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:40.610809 1831 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:40.616222 1523 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:40.618768 1410 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:40.620054 1759 raft_consensus.cc:2781] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:40.620584 1759 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:40.620957 1759 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:40.629088 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:40.628620 1759 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.629768 1603 raft_consensus.cc:2379] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 1: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:09:40.630811 1410 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:40.631994 1648 tablet_bootstrap.cc:654] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:40.633217 1778 raft_consensus.cc:2726] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260808 01:09:40.644418 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 102038us
I20260808 01:09:40.647971 1944 consensus_queue.cc:1035] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:40.650760 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:40.651511 1521 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:09:40.659761 1567 leader_election.cc:341] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:40.660334 1567 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:40.663527 1814 raft_consensus.cc:2726] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260808 01:09:40.668215 1951 log.cc:924] Time spent T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20260808 01:09:40.668624 1816 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.669605 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:40.670979 1816 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:40.671078 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:40.657904 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:40.666319 1759 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:40.671684 1448 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:40.686579 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:09:40.689528 1567 leader_election.cc:341] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:40.689890 1567 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:40.660761 1831 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.694993 1648 tablet_bootstrap.cc:492] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:40.695696 1648 ts_tablet_manager.cc:1397] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.215s user 0.011s sys 0.005s
I20260808 01:09:40.700429 1645 tablet_bootstrap.cc:492] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:40.679816 1883 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:40.706097 1883 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.709216 1883 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:40.685222 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:40.688105 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:40.697675 1875 raft_consensus.cc:2726] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:09:40.684247 1568 leader_election.cc:341] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:09:40.716192 1568 leader_election.cc:341] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:40.717236 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:40.718219 1522 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:09:40.722990 1410 leader_election.cc:341] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:40.701227 1645 ts_tablet_manager.cc:1397] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.338s user 0.011s sys 0.025s
W20260808 01:09:40.738476 1567 leader_election.cc:341] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:40.740217 1567 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:40.756402 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:40.757503 1447 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:40.743144 1814 raft_consensus.cc:2726] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:09:40.736208 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 166889us
I20260808 01:09:40.768033 1490 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:40.778841 1645 raft_consensus.cc:348] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.779095 1966 raft_consensus.cc:2781] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:40.780011 1966 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:40.780484 1966 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:40.782126 1645 raft_consensus.cc:374] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:40.782562 1645 raft_consensus.cc:729] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:40.784163 1645 consensus_queue.cc:260] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.791611 1645 ts_tablet_manager.cc:1428] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.065s user 0.004s sys 0.002s
I20260808 01:09:40.805541 1648 raft_consensus.cc:348] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.806173 1648 raft_consensus.cc:374] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:40.806408 1648 raft_consensus.cc:729] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
W20260808 01:09:40.806632 1963 log.cc:924] Time spent T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260808 01:09:40.809000 1831 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:40.807500 1648 consensus_queue.cc:260] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.810413 1645 tablet_bootstrap.cc:492] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:40.805217 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:40.811311 1523 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:40.822813 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:40.824626 1448 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:40.829879 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:40.830557 1602 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:09:40.835686 1410 leader_election.cc:341] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:40.839784 1648 ts_tablet_manager.cc:1428] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.144s user 0.008s sys 0.000s
I20260808 01:09:40.841303 1490 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:40.847579 1339 catalog_manager.cc:5495] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } }
I20260808 01:09:40.846700 1930 raft_consensus.cc:2781] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:40.855727 1646 tablet_bootstrap.cc:492] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:40.856247 1646 ts_tablet_manager.cc:1397] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.488s user 0.011s sys 0.018s
I20260808 01:09:40.860795 1930 raft_consensus.cc:686] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:40.865404 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:40.869331 1930 consensus_queue.cc:237] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.870227 1944 consensus_queue.cc:1035] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:40.844785 1522 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:40.903844 1567 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:09:40.905663 1814 raft_consensus.cc:2781] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:40.906229 1814 raft_consensus.cc:686] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:40.907126 1814 consensus_queue.cc:237] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.918992 1341 catalog_manager.cc:5495] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:40.913705 1602 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:40.911363 1523 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:40.933846 1521 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:40.955377 1668 consensus_queue.cc:1035] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:40.949088 1646 raft_consensus.cc:348] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.957675 1646 raft_consensus.cc:374] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:40.958041 1646 raft_consensus.cc:729] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:40.958765 1646 consensus_queue.cc:260] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.932945 1966 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.960364 1410 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:40.962178 1778 raft_consensus.cc:2781] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:40.962510 1966 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:40.962841 1778 raft_consensus.cc:686] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:40.963054 1410 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:40.938930 1648 tablet_bootstrap.cc:492] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:40.952935 1341 catalog_manager.cc:5495] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:40.967355 1645 tablet_bootstrap.cc:654] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:40.954378 1489 leader_election.cc:341] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:40.964187 1778 consensus_queue.cc:237] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.979023 1668 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:40.979544 1668 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:40.999329 1668 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:40.997943 1832 raft_consensus.cc:2781] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:41.004309 1832 raft_consensus.cc:686] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:41.005301 1832 consensus_queue.cc:237] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.007174 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:41.008085 1445 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:41.017782 1646 ts_tablet_manager.cc:1428] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.161s user 0.011s sys 0.000s
I20260808 01:09:41.019323 1646 tablet_bootstrap.cc:492] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:41.020785 1875 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:41.021292 1875 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.022341 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:41.043771 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:41.044302 1445 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
W20260808 01:09:41.162199 1489 leader_election.cc:341] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:09:41.166461 1626 debug-util.cc:398] Leaking SignalData structure 0x7b08001ef440 after lost signal to thread 32596
W20260808 01:09:41.180506 1626 debug-util.cc:398] Leaking SignalData structure 0x7b080019b220 after lost signal to thread 1376
I20260808 01:09:41.135787 1448 raft_consensus.cc:2379] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 1: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:09:41.053377 1875 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:41.076377 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:41.150585 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:41.187021 1523 raft_consensus.cc:2379] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 1: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:09:41.189292 1604 raft_consensus.cc:2379] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 1: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:09:41.192169 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:41.198251 1646 tablet_bootstrap.cc:654] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:41.208335 1490 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:41.214627 1822 raft_consensus.cc:2781] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
W20260808 01:09:41.211925 1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800110b60 after lost signal to thread 1473
I20260808 01:09:41.234472 1648 tablet_bootstrap.cc:654] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:41.240175 1490 consensus_peers.cc:487] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Couldn't send request to peer 3fb1a6380d27401a831cfe5f86a4cc9d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:41.227356 1943 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:41.228267 1822 raft_consensus.cc:686] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:41.258441 1943 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:41.271186 1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800152740 after lost signal to thread 1551
W20260808 01:09:41.273418 1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800233080 after lost signal to thread 1629
I20260808 01:09:41.279764 1822 consensus_queue.cc:237] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.290143 1943 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:41.294591 1489 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:41.295457 1930 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:41.295394 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:41.297371 1668 consensus_queue.cc:1035] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:41.299986 1645 tablet_bootstrap.cc:492] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:41.300457 1645 ts_tablet_manager.cc:1397] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.496s user 0.007s sys 0.012s
I20260808 01:09:41.304119 1568 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:41.305154 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:41.307524 1816 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:41.315092 1341 catalog_manager.cc:5495] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:41.317327 1341 catalog_manager.cc:5495] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:41.319412 1339 catalog_manager.cc:5495] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:41.302958 1522 raft_consensus.cc:2379] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 1: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:09:41.327322 1883 consensus_queue.cc:1035] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:41.335178 1930 raft_consensus.cc:2726] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:41.340193 1646 tablet_bootstrap.cc:492] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:41.345937 1646 ts_tablet_manager.cc:1397] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.327s user 0.019s sys 0.022s
I20260808 01:09:41.349097 1646 raft_consensus.cc:348] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.350009 1646 raft_consensus.cc:374] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:41.350502 1646 raft_consensus.cc:729] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:41.361927 1645 raft_consensus.cc:348] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.365749 1645 raft_consensus.cc:374] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:41.366266 1645 raft_consensus.cc:729] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:41.351380 1646 consensus_queue.cc:260] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.370685 1645 consensus_queue.cc:260] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.373499 1982 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:41.380967 1410 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:41.384927 1976 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:41.388450 1645 ts_tablet_manager.cc:1428] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.088s user 0.008s sys 0.001s
I20260808 01:09:41.390398 1646 ts_tablet_manager.cc:1428] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.044s user 0.006s sys 0.002s
I20260808 01:09:41.374090 1982 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.400825 1646 tablet_bootstrap.cc:492] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:41.396222 1645 tablet_bootstrap.cc:492] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:41.423677 1778 consensus_queue.cc:1035] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:41.444895 1648 tablet_bootstrap.cc:492] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:41.445463 1648 ts_tablet_manager.cc:1397] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.507s user 0.027s sys 0.041s
I20260808 01:09:41.434337 1976 raft_consensus.cc:2726] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:41.440202 1816 raft_consensus.cc:2726] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:41.449389 1814 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260808 01:09:41.471323 1930 consensus_queue.cc:1035] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:41.473783 1648 raft_consensus.cc:348] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.476853 1648 raft_consensus.cc:374] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:41.477108 1648 raft_consensus.cc:729] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:41.477679 1648 consensus_queue.cc:260] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.481899 1648 ts_tablet_manager.cc:1428] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.036s user 0.007s sys 0.001s
I20260808 01:09:41.484306 1341 catalog_manager.cc:5495] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:41.497323 1778 consensus_queue.cc:1035] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:41.509073 1648 tablet_bootstrap.cc:492] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:41.519779 1645 tablet_bootstrap.cc:654] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:41.528533 1930 consensus_queue.cc:1035] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:41.569844 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:41.579617 1646 tablet_bootstrap.cc:654] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:41.561072 1778 consensus_queue.cc:1035] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
I20260808 01:09:41.534425 1982 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:41.599620 1648 tablet_bootstrap.cc:654] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:41.609617 1341 catalog_manager.cc:5495] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:41.580741 1814 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
W20260808 01:09:41.705075 1993 log.cc:924] Time spent T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20260808 01:09:41.705493 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:41.736474 1645 tablet_bootstrap.cc:492] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:41.737051 1645 ts_tablet_manager.cc:1397] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.341s user 0.011s sys 0.007s
W20260808 01:09:41.738539 1410 consensus_peers.cc:487] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Couldn't send request to peer a4862d0938c84fc4bd840e4c17acae78. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:41.746948 1645 raft_consensus.cc:348] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.747517 1645 raft_consensus.cc:374] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:41.747725 1645 raft_consensus.cc:729] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:41.740465 1446 raft_consensus.cc:2379] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 1: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
W20260808 01:09:41.752610 1390 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:09:41.772708 1443 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:41.768776 1645 consensus_queue.cc:260] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.789747 1648 tablet_bootstrap.cc:492] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:41.790292 1648 ts_tablet_manager.cc:1397] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.281s user 0.017s sys 0.023s
I20260808 01:09:41.787644 1883 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:41.792281 1645 ts_tablet_manager.cc:1428] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.055s user 0.006s sys 0.000s
I20260808 01:09:41.796273 1645 tablet_bootstrap.cc:492] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:41.795892 1648 raft_consensus.cc:348] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.796666 1648 raft_consensus.cc:374] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:41.796957 1648 raft_consensus.cc:729] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:41.795447 1883 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:41.793632 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 46692.9us
I20260808 01:09:41.799024 1568 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:41.799787 1982 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:41.794555 1646 tablet_bootstrap.cc:492] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:41.802129 1646 ts_tablet_manager.cc:1397] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.409s user 0.019s sys 0.025s
I20260808 01:09:41.804420 1645 tablet_bootstrap.cc:654] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:41.797612 1648 consensus_queue.cc:260] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.821089 1646 raft_consensus.cc:348] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.821729 1646 raft_consensus.cc:374] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:41.821962 1646 raft_consensus.cc:729] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:41.822486 1646 consensus_queue.cc:260] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.823606 1648 ts_tablet_manager.cc:1428] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.033s user 0.009s sys 0.000s
I20260808 01:09:41.833797 1648 tablet_bootstrap.cc:492] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:41.836290 1646 ts_tablet_manager.cc:1428] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.034s user 0.002s sys 0.005s
I20260808 01:09:41.852248 1646 tablet_bootstrap.cc:492] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:41.868355 1883 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:41.884264 1778 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:41.884869 1778 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.885885 1646 tablet_bootstrap.cc:654] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:41.902501 1778 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:41.918599 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:41.919597 1603 raft_consensus.cc:2454] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:41.923564 1645 tablet_bootstrap.cc:492] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:41.891595 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:41.925673 1645 ts_tablet_manager.cc:1397] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.130s user 0.020s sys 0.027s
I20260808 01:09:41.932591 1602 raft_consensus.cc:2454] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:41.940771 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:41.948626 1489 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:41.949225 1524 raft_consensus.cc:2454] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:41.949954 1822 raft_consensus.cc:2781] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:41.950450 1822 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:41.950829 1822 raft_consensus.cc:3037] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:41.959777 1982 raft_consensus.cc:2726] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:41.970296 1944 consensus_queue.cc:1035] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:41.974242 1944 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:41.974813 1944 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:41.986420 1944 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:42.001514 1822 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.005617 1410 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:42.006135 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:42.006683 1646 tablet_bootstrap.cc:492] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:42.007429 1645 raft_consensus.cc:348] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.028029 1645 raft_consensus.cc:374] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:42.028411 1645 raft_consensus.cc:729] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:42.029229 1645 consensus_queue.cc:260] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.027552 1646 ts_tablet_manager.cc:1397] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.181s user 0.022s sys 0.052s
I20260808 01:09:42.009078 1648 tablet_bootstrap.cc:654] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:42.033959 2003 raft_consensus.cc:2781] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:42.034358 2003 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:42.034708 2003 raft_consensus.cc:3037] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:09:42.036012 1489 leader_election.cc:341] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:42.042291 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:42.043031 1598 raft_consensus.cc:3037] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:42.044732 2003 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.047094 2003 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:42.038156 1822 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:42.055783 1648 tablet_bootstrap.cc:492] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:42.056600 1648 ts_tablet_manager.cc:1397] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.226s user 0.011s sys 0.005s
I20260808 01:09:42.056803 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:42.057920 1519 raft_consensus.cc:2379] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 1: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:09:42.058503 1645 ts_tablet_manager.cc:1428] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.133s user 0.015s sys 0.002s
I20260808 01:09:42.035240 1646 raft_consensus.cc:348] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.059821 1646 raft_consensus.cc:374] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:42.060217 1646 raft_consensus.cc:729] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:42.060956 1646 consensus_queue.cc:260] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.061704 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:42.061723 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:42.064845 1646 ts_tablet_manager.cc:1428] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.033s user 0.008s sys 0.000s
I20260808 01:09:42.065979 1646 tablet_bootstrap.cc:492] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:42.065871 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:42.066821 1440 raft_consensus.cc:2379] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 1: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:09:42.071383 1410 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:42.073906 1648 raft_consensus.cc:348] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.074713 1648 raft_consensus.cc:374] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:42.074887 1599 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:42.075062 1648 raft_consensus.cc:729] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:42.075919 1648 consensus_queue.cc:260] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.078167 1648 ts_tablet_manager.cc:1428] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20260808 01:09:42.072824 1832 raft_consensus.cc:2726] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260808 01:09:42.074321 1490 leader_election.cc:341] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:42.077906 1598 raft_consensus.cc:2454] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:42.079689 1490 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:42.081292 1648 tablet_bootstrap.cc:492] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:42.081846 1489 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:09:42.082587 1646 tablet_bootstrap.cc:654] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:42.083066 1930 raft_consensus.cc:2726] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:42.083824 1930 raft_consensus.cc:2781] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:42.084621 1930 raft_consensus.cc:686] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:42.090044 1930 consensus_queue.cc:237] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.105765 1645 tablet_bootstrap.cc:492] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:42.072867 1822 consensus_queue.cc:1035] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
I20260808 01:09:42.115154 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:42.126489 1447 raft_consensus.cc:2379] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 1: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:09:42.132391 1759 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:42.133291 1648 tablet_bootstrap.cc:654] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:42.133044 1759 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.141469 1645 tablet_bootstrap.cc:654] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:42.151409 1883 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:42.167183 1646 tablet_bootstrap.cc:492] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:42.155476 1883 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.179822 1646 ts_tablet_manager.cc:1397] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.114s user 0.010s sys 0.012s
I20260808 01:09:42.187390 1759 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:42.192287 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:42.193782 1883 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:42.202044 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:42.202726 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:42.202987 1447 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:42.206535 1646 raft_consensus.cc:348] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.207363 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:42.207665 1646 raft_consensus.cc:374] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:42.209300 1646 raft_consensus.cc:729] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:42.210310 1646 consensus_queue.cc:260] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.218736 1410 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:42.219921 1759 raft_consensus.cc:2726] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:09:42.208583 1489 leader_election.cc:341] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:42.234786 1490 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:42.230798 1339 catalog_manager.cc:5495] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:42.238705 1883 raft_consensus.cc:2781] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:42.240129 1646 ts_tablet_manager.cc:1428] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.060s user 0.007s sys 0.002s
I20260808 01:09:42.233810 1816 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:42.240834 1883 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:42.242331 1981 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:42.243604 1883 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:42.243769 1816 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.243980 1981 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.246033 1816 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:42.251853 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:42.259258 1645 tablet_bootstrap.cc:492] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:42.261695 1645 ts_tablet_manager.cc:1397] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.156s user 0.015s sys 0.021s
I20260808 01:09:42.263164 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:42.268241 1646 tablet_bootstrap.cc:492] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:42.269591 1648 tablet_bootstrap.cc:492] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:42.270159 1648 ts_tablet_manager.cc:1397] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.189s user 0.015s sys 0.022s
I20260808 01:09:42.278957 1568 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:42.280619 1649 raft_consensus.cc:2726] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:42.281632 1645 raft_consensus.cc:348] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.282473 1645 raft_consensus.cc:374] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:42.282770 1981 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:42.285171 1648 raft_consensus.cc:348] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.286055 1648 raft_consensus.cc:374] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:42.286407 1648 raft_consensus.cc:729] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:42.282771 1645 raft_consensus.cc:729] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:42.287518 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:42.288051 1645 consensus_queue.cc:260] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.289383 1648 consensus_queue.cc:260] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.291302 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:42.294260 1645 ts_tablet_manager.cc:1428] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.032s user 0.006s sys 0.003s
I20260808 01:09:42.276675 1883 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.296547 1648 ts_tablet_manager.cc:1428] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.026s user 0.008s sys 0.000s
I20260808 01:09:42.297793 1568 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:42.299521 1646 tablet_bootstrap.cc:654] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:42.299347 1883 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:42.301750 1645 tablet_bootstrap.cc:492] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:42.302220 1648 tablet_bootstrap.cc:492] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:42.305891 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:42.303282 1981 raft_consensus.cc:2726] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:42.306941 1447 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:42.311641 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:09:42.318398 1489 leader_election.cc:341] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:42.333088 1447 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:42.336278 1648 tablet_bootstrap.cc:654] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:42.339198 1490 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:42.342310 1645 tablet_bootstrap.cc:654] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:42.342695 1822 raft_consensus.cc:2781] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:42.344556 1822 raft_consensus.cc:686] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:42.345577 1822 consensus_queue.cc:237] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.361150 1646 tablet_bootstrap.cc:492] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:42.361806 1646 ts_tablet_manager.cc:1397] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.100s user 0.015s sys 0.015s
I20260808 01:09:42.383072 1646 raft_consensus.cc:348] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.384255 1646 raft_consensus.cc:374] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:42.384660 1646 raft_consensus.cc:729] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:42.385613 1646 consensus_queue.cc:260] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.393397 1646 ts_tablet_manager.cc:1428] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.031s user 0.010s sys 0.001s
I20260808 01:09:42.414077 1646 tablet_bootstrap.cc:492] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:42.429055 1648 tablet_bootstrap.cc:492] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:42.432809 1648 ts_tablet_manager.cc:1397] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.131s user 0.011s sys 0.015s
I20260808 01:09:42.429273 1981 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:42.448882 1943 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:42.449397 1943 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.440748 1816 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:42.450415 1816 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.450739 1981 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.451490 1943 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:42.452555 1816 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:42.452809 1981 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:42.456247 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:42.456568 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:42.457171 1443 raft_consensus.cc:2454] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:42.459820 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:42.460675 1598 raft_consensus.cc:2454] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
W20260808 01:09:42.461222 1568 leader_election.cc:341] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:42.461287 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:42.461884 1568 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:42.462458 1410 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:42.463204 1649 raft_consensus.cc:2781] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:42.463658 1649 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:42.463807 1976 raft_consensus.cc:2781] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:09:42.464159 1649 raft_consensus.cc:3037] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:42.464277 1976 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:42.464730 1976 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:09:42.468474 1567 leader_election.cc:341] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:42.469069 1567 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:42.473151 1816 raft_consensus.cc:2726] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:42.473873 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:42.478216 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:42.478741 1646 tablet_bootstrap.cc:654] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:42.478952 1524 raft_consensus.cc:2454] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
W20260808 01:09:42.480082 1567 leader_election.cc:341] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:42.490897 1648 raft_consensus.cc:348] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.492007 1648 raft_consensus.cc:374] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:42.492295 1648 raft_consensus.cc:729] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:42.487203 1976 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.493937 1648 consensus_queue.cc:260] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.496659 1976 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:42.498122 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:42.500325 1648 ts_tablet_manager.cc:1428] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.067s user 0.007s sys 0.001s
I20260808 01:09:42.500864 1598 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:42.501293 1648 tablet_bootstrap.cc:492] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:42.501530 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:42.502143 1524 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:42.510737 1339 catalog_manager.cc:5495] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:42.499189 1649 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.514451 1649 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:42.516206 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:42.516957 1648 tablet_bootstrap.cc:654] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:42.516849 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:42.517496 1447 raft_consensus.cc:3037] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:09:42.518756 1567 leader_election.cc:341] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:42.520816 1645 tablet_bootstrap.cc:492] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:42.521358 1645 ts_tablet_manager.cc:1397] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.220s user 0.012s sys 0.014s
I20260808 01:09:42.526436 1598 raft_consensus.cc:2454] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:09:42.533274 1524 raft_consensus.cc:2454] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:09:42.534885 1447 raft_consensus.cc:2454] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:42.542096 1410 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:42.543884 1568 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:42.543972 1832 raft_consensus.cc:2781] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:09:42.545315 1832 raft_consensus.cc:686] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:42.546208 1832 consensus_queue.cc:237] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.547572 1645 raft_consensus.cc:348] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.549317 1646 tablet_bootstrap.cc:492] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:42.549782 1646 ts_tablet_manager.cc:1397] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.136s user 0.015s sys 0.002s
I20260808 01:09:42.548362 1645 raft_consensus.cc:374] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:42.550415 1645 raft_consensus.cc:729] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:42.551090 1645 consensus_queue.cc:260] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.551807 1822 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:42.552321 1822 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.553326 1645 ts_tablet_manager.cc:1428] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20260808 01:09:42.554303 1645 tablet_bootstrap.cc:492] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:42.554497 1822 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:42.557934 1646 raft_consensus.cc:348] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.560597 1646 raft_consensus.cc:374] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:42.560851 1646 raft_consensus.cc:729] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:42.561448 1646 consensus_queue.cc:260] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.552716 1649 raft_consensus.cc:2781] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:42.557456 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:42.563161 1602 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:42.563432 1599 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:42.564100 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:42.564551 1489 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:42.565574 2004 raft_consensus.cc:2781] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260808 01:09:42.565685 1490 leader_election.cc:341] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:42.566010 2004 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:42.567236 2004 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:42.573669 1649 raft_consensus.cc:686] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:42.582244 1646 ts_tablet_manager.cc:1428] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.032s user 0.002s sys 0.004s
I20260808 01:09:42.581435 1649 consensus_queue.cc:237] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.594643 1646 tablet_bootstrap.cc:492] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:42.598335 2019 consensus_queue.cc:1035] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:42.623750 1339 catalog_manager.cc:5495] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 2, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:42.626796 2004 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.653167 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:42.654050 1602 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:42.657373 2004 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:42.657414 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:42.662222 1648 tablet_bootstrap.cc:492] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:42.662846 1648 ts_tablet_manager.cc:1397] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.162s user 0.016s sys 0.044s
W20260808 01:09:42.666424 1490 leader_election.cc:341] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:42.669209 1648 raft_consensus.cc:348] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.670504 1648 raft_consensus.cc:374] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:42.670833 1648 raft_consensus.cc:729] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:42.684878 1645 tablet_bootstrap.cc:654] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:42.680128 1648 consensus_queue.cc:260] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.678943 1339 catalog_manager.cc:5495] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:42.682317 2004 consensus_queue.cc:1035] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
I20260808 01:09:42.721275 1648 ts_tablet_manager.cc:1428] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.058s user 0.018s sys 0.003s
I20260808 01:09:42.679313 1602 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:42.754930 1489 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:09:42.764232 1648 tablet_bootstrap.cc:492] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:42.758188 1930 raft_consensus.cc:2781] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:42.768226 1930 raft_consensus.cc:686] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:42.796936 1646 tablet_bootstrap.cc:654] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:42.796936 1648 tablet_bootstrap.cc:654] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:42.794544 1930 consensus_queue.cc:237] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.830072 1649 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:09:42.832294 1390 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:09:42.830801 1649 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.848096 1649 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:42.851321 1645 tablet_bootstrap.cc:492] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:42.841810 1944 consensus_queue.cc:1035] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260808 01:09:42.856882 1944 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:42.857370 1944 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.859027 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:42.853441 1645 ts_tablet_manager.cc:1397] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.299s user 0.018s sys 0.053s
I20260808 01:09:42.853495 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:42.863159 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:42.868949 1944 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:09:42.871132 1490 leader_election.cc:341] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:42.868621 1645 raft_consensus.cc:348] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.872370 1645 raft_consensus.cc:374] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:42.872664 1645 raft_consensus.cc:729] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:42.874451 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:09:42.874702 1568 leader_election.cc:341] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:42.859730 1522 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:42.875140 1599 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:42.876355 1567 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:09:42.877184 1982 raft_consensus.cc:2781] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:42.876987 1489 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:09:42.877609 1982 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:42.878007 1982 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:42.879237 2004 raft_consensus.cc:2781] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:42.879773 2004 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:42.880220 2004 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:42.891754 1645 consensus_queue.cc:260] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.899178 1646 tablet_bootstrap.cc:492] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
W20260808 01:09:42.901779 1489 consensus_peers.cc:487] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Couldn't send request to peer a4862d0938c84fc4bd840e4c17acae78. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:42.908239 1646 ts_tablet_manager.cc:1397] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.314s user 0.013s sys 0.011s
I20260808 01:09:42.902873 2004 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.911170 1646 raft_consensus.cc:348] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.911885 1646 raft_consensus.cc:374] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:42.912149 1646 raft_consensus.cc:729] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:42.906785 1982 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.914000 1982 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:42.905704 1339 catalog_manager.cc:5495] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:42.914206 1648 tablet_bootstrap.cc:492] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:42.920231 1648 ts_tablet_manager.cc:1397] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.161s user 0.024s sys 0.022s
I20260808 01:09:42.910923 1645 ts_tablet_manager.cc:1428] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.050s user 0.006s sys 0.002s
I20260808 01:09:42.915761 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:42.916895 2004 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:42.917299 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:42.925307 1522 raft_consensus.cc:2379] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 1: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:09:42.918701 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:42.927559 1598 raft_consensus.cc:2379] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 1: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
W20260808 01:09:42.930583 1568 leader_election.cc:341] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:42.931547 1567 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:42.932281 1982 raft_consensus.cc:2726] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:09:42.933840 1645 tablet_bootstrap.cc:492] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:42.912902 1646 consensus_queue.cc:260] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.941021 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:42.945541 1645 tablet_bootstrap.cc:654] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:42.949297 1490 leader_election.cc:341] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:42.951398 1490 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:42.952255 1648 raft_consensus.cc:348] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.953034 1648 raft_consensus.cc:374] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:42.953316 1648 raft_consensus.cc:729] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:42.953979 1648 consensus_queue.cc:260] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:42.955003 1930 raft_consensus.cc:2726] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:09:42.964389 1648 ts_tablet_manager.cc:1428] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20260808 01:09:42.969659 1648 tablet_bootstrap.cc:492] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:42.980329 1646 ts_tablet_manager.cc:1428] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.072s user 0.009s sys 0.000s
I20260808 01:09:42.981400 1646 tablet_bootstrap.cc:492] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:42.999173 1648 tablet_bootstrap.cc:654] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:43.006671 1645 tablet_bootstrap.cc:492] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:43.006997 1646 tablet_bootstrap.cc:654] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:43.007265 1645 ts_tablet_manager.cc:1397] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.074s user 0.013s sys 0.016s
I20260808 01:09:43.022518 1645 raft_consensus.cc:348] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.023346 1645 raft_consensus.cc:374] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:43.023617 1645 raft_consensus.cc:729] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:43.024304 1645 consensus_queue.cc:260] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.029611 1648 tablet_bootstrap.cc:492] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:43.030406 1648 ts_tablet_manager.cc:1397] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.061s user 0.017s sys 0.001s
I20260808 01:09:43.030644 1645 ts_tablet_manager.cc:1428] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260808 01:09:43.034046 1645 tablet_bootstrap.cc:492] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:43.034323 1976 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:43.033447 1648 raft_consensus.cc:348] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.035192 1648 raft_consensus.cc:374] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:43.035528 1648 raft_consensus.cc:729] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:43.034812 1976 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.036247 1648 consensus_queue.cc:260] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.041079 1646 tablet_bootstrap.cc:492] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:43.041579 1646 ts_tablet_manager.cc:1397] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.060s user 0.012s sys 0.003s
I20260808 01:09:43.043363 1648 ts_tablet_manager.cc:1428] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260808 01:09:43.048205 1976 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:43.050616 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:43.051697 1522 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:43.053953 2004 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:43.053432 1646 raft_consensus.cc:348] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.054939 1646 raft_consensus.cc:374] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:43.055285 1646 raft_consensus.cc:729] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:43.054714 2004 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.057252 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:43.059370 1648 tablet_bootstrap.cc:492] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:43.059599 1410 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:43.061060 1646 consensus_queue.cc:260] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.062964 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:43.058571 2004 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:43.070430 1646 ts_tablet_manager.cc:1428] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.029s user 0.001s sys 0.008s
W20260808 01:09:43.065353 1489 leader_election.cc:341] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:43.067297 1832 raft_consensus.cc:2781] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:43.100841 1832 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:43.101562 1832 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:43.102600 1646 tablet_bootstrap.cc:492] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
W20260808 01:09:43.069085 1410 leader_election.cc:341] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:43.092689 1645 tablet_bootstrap.cc:654] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:43.108240 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:43.124583 1832 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.129911 1832 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:43.130911 1648 tablet_bootstrap.cc:654] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:43.139119 1646 tablet_bootstrap.cc:654] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:43.133994 1976 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:43.144714 1976 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.148749 1976 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:43.153381 1645 tablet_bootstrap.cc:492] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:43.154052 1645 ts_tablet_manager.cc:1397] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.122s user 0.012s sys 0.013s
W20260808 01:09:43.156342 1490 leader_election.cc:341] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:43.156942 1490 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:43.159076 2004 raft_consensus.cc:2726] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:43.163147 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:43.164250 1522 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:43.163235 1645 raft_consensus.cc:348] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.165143 1645 raft_consensus.cc:374] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:43.165505 1645 raft_consensus.cc:729] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:43.166484 1646 tablet_bootstrap.cc:492] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:43.167330 1646 ts_tablet_manager.cc:1397] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.023s
I20260808 01:09:43.167227 1599 raft_consensus.cc:1261] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260808 01:09:43.166468 1645 consensus_queue.cc:260] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.172641 1645 ts_tablet_manager.cc:1428] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.018s user 0.009s sys 0.004s
I20260808 01:09:43.172384 1982 consensus_queue.cc:1035] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:43.173987 1645 tablet_bootstrap.cc:492] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:43.179601 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:43.180892 1598 raft_consensus.cc:2454] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:43.185046 1524 raft_consensus.cc:1261] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260808 01:09:43.185918 1648 tablet_bootstrap.cc:492] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:43.186508 1648 ts_tablet_manager.cc:1397] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.127s user 0.005s sys 0.012s
I20260808 01:09:43.186604 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:43.190237 1645 tablet_bootstrap.cc:654] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:43.190140 1648 raft_consensus.cc:348] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.190807 1648 raft_consensus.cc:374] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:43.191172 1648 raft_consensus.cc:729] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:43.192094 1648 consensus_queue.cc:260] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.196818 1943 consensus_queue.cc:1035] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260808 01:09:43.201835 1410 leader_election.cc:341] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:43.197021 1646 raft_consensus.cc:348] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.202643 1646 raft_consensus.cc:374] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:43.202986 1646 raft_consensus.cc:729] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:43.207669 1410 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:43.208499 1976 raft_consensus.cc:2781] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:43.208895 1976 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:43.209297 1976 raft_consensus.cc:3037] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:43.210541 1648 ts_tablet_manager.cc:1428] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.024s user 0.009s sys 0.000s
I20260808 01:09:43.211819 1648 tablet_bootstrap.cc:492] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:43.202131 1759 consensus_queue.cc:1035] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:43.225119 1832 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:43.231691 1976 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.234797 1518 raft_consensus.cc:3037] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:43.237970 1648 tablet_bootstrap.cc:654] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:43.203799 1646 consensus_queue.cc:260] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.225982 1832 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.251730 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:43.268493 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:43.269388 1603 raft_consensus.cc:3037] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:09:43.269527 1410 leader_election.cc:341] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:43.272934 1646 ts_tablet_manager.cc:1428] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.105s user 0.009s sys 0.000s
I20260808 01:09:43.234874 1976 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:43.276127 1875 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:43.285353 1875 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.288398 1875 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:43.290886 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:43.291563 1524 raft_consensus.cc:2454] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:43.291397 1930 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:43.292490 1522 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:43.295207 1832 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:43.292932 1567 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:43.297132 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:43.298390 1599 raft_consensus.cc:2379] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 1: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:09:43.299263 1646 tablet_bootstrap.cc:492] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:43.313195 1982 raft_consensus.cc:2781] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:43.315392 1982 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:43.316064 1982 raft_consensus.cc:3037] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:43.316377 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:43.325922 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:43.328743 1410 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:43.329420 1645 tablet_bootstrap.cc:492] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:43.329751 1832 raft_consensus.cc:2781] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
W20260808 01:09:43.330251 1410 leader_election.cc:341] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:43.330458 1645 ts_tablet_manager.cc:1397] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.157s user 0.014s sys 0.024s
I20260808 01:09:43.336869 1410 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:43.342705 1943 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:43.345533 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:43.330422 1832 raft_consensus.cc:686] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:43.350697 1645 raft_consensus.cc:348] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.351480 1645 raft_consensus.cc:374] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:43.351889 1645 raft_consensus.cc:729] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:43.352732 1645 consensus_queue.cc:260] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.362463 1649 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:43.362996 1649 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.369946 1649 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:43.377948 1645 ts_tablet_manager.cc:1428] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.047s user 0.006s sys 0.001s
I20260808 01:09:43.377102 1598 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:43.387562 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:43.388164 1645 tablet_bootstrap.cc:492] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:43.379470 2004 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:43.392683 2004 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.405292 1646 tablet_bootstrap.cc:654] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:43.405169 1568 leader_election.cc:341] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:43.420831 1982 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.376749 1603 raft_consensus.cc:2454] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:43.431854 1649 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:43.432497 1649 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.439996 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:43.441210 1443 raft_consensus.cc:2454] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:43.445355 2004 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:43.447940 1649 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:43.349175 1832 consensus_queue.cc:237] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.453742 1490 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:43.451092 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:43.469961 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:43.471868 1524 raft_consensus.cc:3037] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:43.468667 1982 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:43.471268 1410 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:43.468922 1944 raft_consensus.cc:2781] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:43.484315 2047 raft_consensus.cc:2781] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:43.485271 2047 raft_consensus.cc:686] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:43.486933 1524 raft_consensus.cc:2454] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:43.488207 1567 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:43.486133 2047 consensus_queue.cc:237] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.489161 1982 raft_consensus.cc:2781] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:43.489876 1982 raft_consensus.cc:686] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:43.491093 1982 consensus_queue.cc:237] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.491925 1930 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:43.492683 1930 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:43.497318 1567 leader_election.cc:341] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:43.483667 1944 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:43.505849 1944 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:43.511493 1649 consensus_queue.cc:1035] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:43.518218 1943 raft_consensus.cc:2726] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:43.518929 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:43.528298 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:43.532641 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:43.553267 1930 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:43.581274 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:43.534431 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:43.602419 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:43.603953 1648 tablet_bootstrap.cc:492] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:43.604557 1648 ts_tablet_manager.cc:1397] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.393s user 0.047s sys 0.054s
I20260808 01:09:43.597357 2047 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:09:43.631705 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 48972.2us
W20260808 01:09:43.632892 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 54175.5us
W20260808 01:09:43.633260 1568 leader_election.cc:341] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:09:43.640224 1568 leader_election.cc:341] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:43.640676 1568 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:43.631258 2047 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.641430 1649 raft_consensus.cc:2726] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:43.642560 1568 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:43.643277 1649 raft_consensus.cc:2726] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:09:43.645820 1489 leader_election.cc:341] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:43.651811 1490 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:43.652372 1648 raft_consensus.cc:348] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.653165 1648 raft_consensus.cc:374] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:43.653484 1648 raft_consensus.cc:729] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:43.654048 1648 consensus_queue.cc:260] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.632140 1944 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.655531 1930 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260808 01:09:43.669292 1645 tablet_bootstrap.cc:654] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:43.656438 2057 raft_consensus.cc:2726] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:43.674513 1944 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:43.683908 1648 ts_tablet_manager.cc:1428] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.079s user 0.007s sys 0.001s
W20260808 01:09:43.684072 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 44296.1us
I20260808 01:09:43.694826 2047 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:43.697667 1648 tablet_bootstrap.cc:492] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:43.702756 1341 catalog_manager.cc:5495] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:43.716434 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:43.717705 1522 raft_consensus.cc:2379] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 1: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:09:43.762024 1646 tablet_bootstrap.cc:492] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:43.762993 1646 ts_tablet_manager.cc:1397] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.481s user 0.030s sys 0.025s
I20260808 01:09:43.777092 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:43.781631 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:43.784698 1598 raft_consensus.cc:2379] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 1: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:09:43.789413 1646 raft_consensus.cc:348] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.792466 1646 raft_consensus.cc:374] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:43.792735 1646 raft_consensus.cc:729] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:43.799003 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:43.799991 1442 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:43.879197 1442 raft_consensus.cc:2454] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:43.893164 1646 consensus_queue.cc:260] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.896616 1645 tablet_bootstrap.cc:492] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:43.897132 1645 ts_tablet_manager.cc:1397] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.509s user 0.018s sys 0.017s
I20260808 01:09:43.897272 1646 ts_tablet_manager.cc:1428] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.132s user 0.019s sys 0.006s
I20260808 01:09:43.898350 1646 tablet_bootstrap.cc:492] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:43.910697 1646 tablet_bootstrap.cc:654] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:43.938035 1646 tablet_bootstrap.cc:492] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:43.938722 1646 ts_tablet_manager.cc:1397] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.016s
I20260808 01:09:43.942384 1646 raft_consensus.cc:348] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:43.949580 1646 raft_consensus.cc:374] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:43.895695 1648 tablet_bootstrap.cc:654] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:43.957880 1489 leader_election.cc:341] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:43.961176 1649 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:43.961783 1649 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:44.045338 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 84732.8us
I20260808 01:09:44.046548 1646 raft_consensus.cc:729] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:44.048779 1649 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:44.075681 1646 consensus_queue.cc:260] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.078228 1646 ts_tablet_manager.cc:1428] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.139s user 0.005s sys 0.006s
I20260808 01:09:44.075519 1645 raft_consensus.cc:348] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.081364 1645 raft_consensus.cc:374] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:44.081995 1645 raft_consensus.cc:729] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:44.083072 1645 consensus_queue.cc:260] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.089131 1646 tablet_bootstrap.cc:492] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:44.093901 1645 ts_tablet_manager.cc:1428] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.196s user 0.009s sys 0.010s
I20260808 01:09:44.096696 1645 tablet_bootstrap.cc:492] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:44.102447 1646 tablet_bootstrap.cc:654] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:44.123593 1648 tablet_bootstrap.cc:492] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
W20260808 01:09:44.131927 1390 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:09:44.133286 1944 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:44.153003 1648 ts_tablet_manager.cc:1397] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.456s user 0.015s sys 0.009s
W20260808 01:09:44.142289 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 161947us
I20260808 01:09:44.146217 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:44.156886 1944 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.161226 1648 raft_consensus.cc:348] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.162307 1648 raft_consensus.cc:374] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:44.162662 1648 raft_consensus.cc:729] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:44.163523 1648 consensus_queue.cc:260] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.159674 1944 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:44.182345 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:44.195585 1339 catalog_manager.cc:5495] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:44.197770 1648 ts_tablet_manager.cc:1428] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.044s user 0.010s sys 0.000s
I20260808 01:09:44.207744 1597 raft_consensus.cc:2379] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 1: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:09:44.215297 1447 raft_consensus.cc:3037] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:44.235816 1410 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:09:44.243415 1568 leader_election.cc:341] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:44.251731 1649 consensus_queue.cc:1035] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:44.248981 2074 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:44.257691 1648 tablet_bootstrap.cc:492] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:44.286863 1490 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:44.292068 1646 tablet_bootstrap.cc:492] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:44.292743 1646 ts_tablet_manager.cc:1397] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.204s user 0.022s sys 0.004s
I20260808 01:09:44.308369 2080 raft_consensus.cc:2781] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:44.309420 2080 raft_consensus.cc:686] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:44.322432 2073 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:44.327049 1522 raft_consensus.cc:3037] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:44.311189 2080 consensus_queue.cc:237] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.323292 2073 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.335302 1646 raft_consensus.cc:348] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.336238 1646 raft_consensus.cc:374] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:44.336556 1646 raft_consensus.cc:729] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:44.337280 1646 consensus_queue.cc:260] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.356621 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:44.357334 1519 raft_consensus.cc:2454] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:44.365118 1648 tablet_bootstrap.cc:654] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:44.366539 1646 ts_tablet_manager.cc:1428] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.073s user 0.007s sys 0.002s
I20260808 01:09:44.378337 1645 tablet_bootstrap.cc:654] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:44.378589 2081 consensus_queue.cc:1035] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:44.375517 2074 raft_consensus.cc:2726] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:44.384111 2073 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:44.389065 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:44.398343 1646 tablet_bootstrap.cc:492] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:44.401675 1490 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:09:44.406351 1410 consensus_peers.cc:487] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Couldn't send request to peer a4862d0938c84fc4bd840e4c17acae78. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:44.406708 1567 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:09:44.409042 1649 raft_consensus.cc:2781] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:44.409400 1649 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:44.409729 1649 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:44.412477 1759 consensus_queue.cc:1035] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:44.424566 1875 consensus_queue.cc:1035] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260808 01:09:44.405078 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:44.448364 2070 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:44.449321 2070 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.453919 1646 tablet_bootstrap.cc:654] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:44.456475 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 44817.7us
I20260808 01:09:44.460561 2062 consensus_queue.cc:1035] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260808 01:09:44.461741 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:44.465387 1649 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.443221 1822 raft_consensus.cc:2726] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:09:44.473498 1567 leader_election.cc:341] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:44.484871 1649 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:44.485387 2070 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:44.492159 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:44.493315 1520 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:44.496713 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:09:44.507217 1568 leader_election.cc:341] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:44.507967 1568 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:44.520007 1982 raft_consensus.cc:2726] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:44.538261 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:44.555289 1341 catalog_manager.cc:5495] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
W20260808 01:09:44.584209 1568 leader_election.cc:341] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:44.600957 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:44.611120 1339 catalog_manager.cc:5495] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:44.679481 2078 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:44.680200 2078 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.685210 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:44.686084 1597 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:44.687573 1489 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:44.688352 1648 tablet_bootstrap.cc:492] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:44.688419 2056 raft_consensus.cc:2781] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:09:44.689059 1648 ts_tablet_manager.cc:1397] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.445s user 0.011s sys 0.042s
I20260808 01:09:44.689241 2056 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:44.689762 2056 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:44.702827 1645 tablet_bootstrap.cc:492] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:44.703583 1645 ts_tablet_manager.cc:1397] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.607s user 0.039s sys 0.057s
I20260808 01:09:44.683614 2078 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:44.700197 1444 raft_consensus.cc:2379] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 1: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:09:44.701877 1520 raft_consensus.cc:2454] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:44.742894 1648 raft_consensus.cc:348] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.743816 1648 raft_consensus.cc:374] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:44.744131 1648 raft_consensus.cc:729] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:44.744849 1648 consensus_queue.cc:260] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.747143 1567 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:09:44.749502 1982 raft_consensus.cc:2781] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:44.752369 1982 raft_consensus.cc:686] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:44.753645 1982 consensus_queue.cc:237] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.766595 1645 raft_consensus.cc:348] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.767575 1645 raft_consensus.cc:374] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:44.769132 1645 raft_consensus.cc:729] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:44.769887 1568 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:44.769917 1645 consensus_queue.cc:260] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.776714 1648 ts_tablet_manager.cc:1428] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.087s user 0.005s sys 0.005s
I20260808 01:09:44.781677 1648 tablet_bootstrap.cc:492] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:44.791638 1645 ts_tablet_manager.cc:1428] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.088s user 0.008s sys 0.000s
I20260808 01:09:44.782791 2085 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:44.810789 1645 tablet_bootstrap.cc:492] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
W20260808 01:09:44.825044 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 75762.4us
I20260808 01:09:44.829259 1646 tablet_bootstrap.cc:492] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:44.829959 1646 ts_tablet_manager.cc:1397] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.432s user 0.026s sys 0.033s
I20260808 01:09:44.854497 2078 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:44.859108 2078 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.862897 1646 raft_consensus.cc:348] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.863757 1646 raft_consensus.cc:374] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:44.864079 1646 raft_consensus.cc:729] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:44.864748 1646 consensus_queue.cc:260] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.854496 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:44.885532 1646 ts_tablet_manager.cc:1428] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.055s user 0.008s sys 0.000s
I20260808 01:09:44.891510 1648 tablet_bootstrap.cc:654] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:44.890810 1442 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:44.903613 2078 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:09:44.922266 2089 log.cc:924] Time spent T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.229s user 0.006s sys 0.000s
I20260808 01:09:44.927810 1646 tablet_bootstrap.cc:492] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:44.887964 1822 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:44.934015 1822 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:44.936846 1822 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:44.966122 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:44.973644 2085 raft_consensus.cc:2726] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:09:44.976878 2101 log.cc:924] Time spent T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.065s user 0.006s sys 0.000s
I20260808 01:09:44.988235 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:09:44.988396 2102 log.cc:924] Time spent T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.086s user 0.000s sys 0.004s
I20260808 01:09:44.989975 1645 tablet_bootstrap.cc:654] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:44.888358 2056 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:44.998886 1489 leader_election.cc:341] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:45.004570 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:45.005949 2056 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:45.000283 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:45.010571 1604 raft_consensus.cc:2454] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
W20260808 01:09:45.021237 1490 leader_election.cc:341] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:45.024649 1490 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:45.024771 1489 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:45.026520 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:45.032660 1445 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:45.016219 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:45.040083 1944 raft_consensus.cc:2726] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:45.043144 2027 raft_consensus.cc:2781] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:45.043823 2027 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:45.044346 2027 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:09:45.048691 1490 leader_election.cc:341] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:45.052903 2077 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:45.022246 2056 consensus_queue.cc:1035] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.027s
I20260808 01:09:45.063742 1445 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:09:45.048291 1596 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:45.072410 1930 consensus_queue.cc:1035] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260808 01:09:45.092012 1648 tablet_bootstrap.cc:492] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:45.092631 1648 ts_tablet_manager.cc:1397] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.311s user 0.030s sys 0.008s
I20260808 01:09:45.053720 2077 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.116330 1646 tablet_bootstrap.cc:654] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:45.119827 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 62304us
I20260808 01:09:45.127264 2077 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:45.130053 1490 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:45.135965 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:45.136658 1603 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:45.137965 1822 raft_consensus.cc:2781] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:09:45.138579 1822 raft_consensus.cc:686] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:45.139683 1822 consensus_queue.cc:237] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.147166 1410 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:45.147790 1759 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:45.149318 1759 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:45.149631 1759 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:45.122454 2027 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.169680 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:45.174948 2027 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:45.180462 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:45.181284 1599 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:45.179566 1648 raft_consensus.cc:348] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.181871 1648 raft_consensus.cc:374] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:45.182215 1648 raft_consensus.cc:729] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:45.182911 1648 consensus_queue.cc:260] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.184454 1521 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:45.192406 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:45.183691 1340 catalog_manager.cc:5495] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:45.193661 1645 tablet_bootstrap.cc:492] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:45.194641 1645 ts_tablet_manager.cc:1397] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.391s user 0.027s sys 0.021s
I20260808 01:09:45.197757 1759 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.201515 1759 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:09:45.208761 1568 consensus_peers.cc:487] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 -> Peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Couldn't send request to peer 3fb1a6380d27401a831cfe5f86a4cc9d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:45.214613 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:45.215327 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:45.216476 1598 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:45.230602 1648 ts_tablet_manager.cc:1428] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.138s user 0.005s sys 0.004s
I20260808 01:09:45.234135 1645 raft_consensus.cc:348] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.234760 1645 raft_consensus.cc:374] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:45.234987 1645 raft_consensus.cc:729] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
W20260808 01:09:45.235955 1490 leader_election.cc:341] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:45.242013 1648 tablet_bootstrap.cc:492] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:45.253383 1645 consensus_queue.cc:260] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:45.259573 1390 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:09:45.278429 1645 ts_tablet_manager.cc:1428] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.083s user 0.012s sys 0.008s
I20260808 01:09:45.279580 1645 tablet_bootstrap.cc:492] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:45.283990 1646 tablet_bootstrap.cc:492] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:45.284468 1646 ts_tablet_manager.cc:1397] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.357s user 0.018s sys 0.012s
I20260808 01:09:45.294066 1648 tablet_bootstrap.cc:654] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:45.370546 1599 raft_consensus.cc:2454] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:45.370795 1646 raft_consensus.cc:348] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.371501 1646 raft_consensus.cc:374] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:45.374570 1596 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:09:45.378391 1645 tablet_bootstrap.cc:654] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:45.371789 1646 raft_consensus.cc:729] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:45.388423 1646 consensus_queue.cc:260] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.393325 1646 ts_tablet_manager.cc:1428] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.109s user 0.012s sys 0.004s
I20260808 01:09:45.394373 1646 tablet_bootstrap.cc:492] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:45.379866 1598 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:45.401496 1648 tablet_bootstrap.cc:492] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:45.402055 1648 ts_tablet_manager.cc:1397] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.160s user 0.008s sys 0.012s
I20260808 01:09:45.402309 1489 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:09:45.407361 2062 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:45.408859 1648 raft_consensus.cc:348] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.409624 1648 raft_consensus.cc:374] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:45.409938 1648 raft_consensus.cc:729] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:45.408174 2062 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.410898 1648 consensus_queue.cc:260] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.414055 1410 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:45.414907 1759 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:45.415307 1759 raft_consensus.cc:686] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:45.416131 1759 consensus_queue.cc:237] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.416627 2056 raft_consensus.cc:2781] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:45.416976 1645 tablet_bootstrap.cc:492] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:45.417440 1645 ts_tablet_manager.cc:1397] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.138s user 0.016s sys 0.004s
I20260808 01:09:45.425928 2056 raft_consensus.cc:686] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:45.427479 2056 consensus_queue.cc:237] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.430387 2062 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:45.433845 1648 ts_tablet_manager.cc:1428] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.031s user 0.007s sys 0.001s
I20260808 01:09:45.434321 2073 consensus_queue.cc:1035] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:45.436239 1648 tablet_bootstrap.cc:492] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:45.434492 1645 raft_consensus.cc:348] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.441947 1645 raft_consensus.cc:374] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:45.442425 1645 raft_consensus.cc:729] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:45.457363 1645 consensus_queue.cc:260] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.483007 1649 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:45.483686 1649 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.486210 1649 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:45.492107 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:45.485065 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:45.500927 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:45.501503 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:45.502599 1522 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:45.504259 2078 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:45.506292 1567 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:45.507016 1646 tablet_bootstrap.cc:654] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:45.508922 1645 ts_tablet_manager.cc:1428] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.091s user 0.004s sys 0.006s
I20260808 01:09:45.505043 2078 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.509539 1982 raft_consensus.cc:2781] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:45.514406 1982 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:45.514744 1982 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:09:45.521564 1568 leader_election.cc:341] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:45.524199 1596 raft_consensus.cc:2379] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 1: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:09:45.530936 1648 tablet_bootstrap.cc:654] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:45.532305 1410 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:45.540865 2078 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:45.541872 2059 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:45.554106 1645 tablet_bootstrap.cc:492] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:45.558525 1598 raft_consensus.cc:1261] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260808 01:09:45.559048 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:45.545307 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:09:45.565587 1490 leader_election.cc:341] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:45.583695 2059 raft_consensus.cc:2726] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:45.594992 1342 catalog_manager.cc:5495] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 2, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 2 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:45.605922 1982 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.625137 1982 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:45.625324 1599 raft_consensus.cc:2379] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 1: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:09:45.651342 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:45.652724 1525 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:45.606283 2078 consensus_queue.cc:1035] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.049s
I20260808 01:09:45.663204 1489 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:45.631096 1982 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:45.693802 1444 raft_consensus.cc:1261] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260808 01:09:45.681602 1982 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.712625 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:45.720183 1759 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:45.720806 1759 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.722995 1759 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:45.699401 1982 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:09:45.728717 1567 leader_election.cc:341] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:45.772552 1648 tablet_bootstrap.cc:492] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:45.773307 1648 ts_tablet_manager.cc:1397] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.337s user 0.027s sys 0.025s
I20260808 01:09:45.741092 2056 consensus_queue.cc:1035] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.044s
I20260808 01:09:45.760545 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:45.805593 1646 tablet_bootstrap.cc:492] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:45.806349 1646 ts_tablet_manager.cc:1397] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.412s user 0.044s sys 0.045s
I20260808 01:09:45.806546 1645 tablet_bootstrap.cc:654] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:45.834087 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:45.851075 1648 raft_consensus.cc:348] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.852489 1648 raft_consensus.cc:374] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:45.852905 1648 raft_consensus.cc:729] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:45.882422 1646 raft_consensus.cc:348] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.883451 1646 raft_consensus.cc:374] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:45.853596 1648 consensus_queue.cc:260] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.888675 1524 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:45.883852 1646 raft_consensus.cc:729] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
W20260808 01:09:45.884032 1568 leader_election.cc:341] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:45.892491 1646 consensus_queue.cc:260] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.893522 1568 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:45.894323 1649 raft_consensus.cc:2726] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:45.898835 1648 ts_tablet_manager.cc:1428] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.125s user 0.006s sys 0.003s
I20260808 01:09:45.901352 1646 ts_tablet_manager.cc:1428] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.095s user 0.006s sys 0.004s
I20260808 01:09:45.911650 2027 consensus_queue.cc:1035] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260808 01:09:45.920660 2077 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260808 01:09:45.934731 1522 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:45.938714 1525 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:45.939937 2078 raft_consensus.cc:2726] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:45.942764 1645 tablet_bootstrap.cc:492] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:45.943545 1645 ts_tablet_manager.cc:1397] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.390s user 0.030s sys 0.016s
I20260808 01:09:45.948107 1645 raft_consensus.cc:348] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.950232 1645 raft_consensus.cc:374] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:45.950664 1339 catalog_manager.cc:5495] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:45.953403 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:45.954510 1597 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:45.954984 1410 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:45.955811 2128 raft_consensus.cc:2781] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:45.956457 2128 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:45.950683 1645 raft_consensus.cc:729] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:45.956895 2128 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:45.957489 1645 consensus_queue.cc:260] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:45.960321 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:45.961396 1439 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:45.961877 1645 ts_tablet_manager.cc:1428] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20260808 01:09:45.963078 1645 tablet_bootstrap.cc:492] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:45.972483 1648 tablet_bootstrap.cc:492] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:45.987552 1646 tablet_bootstrap.cc:492] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
W20260808 01:09:46.037812 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 110990us
I20260808 01:09:46.038669 1645 tablet_bootstrap.cc:654] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:46.041102 1875 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:46.042774 1646 tablet_bootstrap.cc:654] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:46.044926 1567 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:46.052201 1875 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.053297 1649 raft_consensus.cc:2781] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:46.053854 1649 raft_consensus.cc:686] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:46.054688 1649 consensus_queue.cc:237] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:46.045805 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 101586us
I20260808 01:09:46.064961 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:09:46.073700 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 118814us
I20260808 01:09:46.055505 1875 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:46.121379 1340 catalog_manager.cc:5495] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:46.122603 1648 tablet_bootstrap.cc:654] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
W20260808 01:09:46.127285 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 68439.3us
I20260808 01:09:46.131649 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:46.132560 1517 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:46.150276 1759 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:46.150674 1567 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:46.170946 1645 tablet_bootstrap.cc:492] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:46.171663 1645 ts_tablet_manager.cc:1397] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.209s user 0.043s sys 0.072s
I20260808 01:09:46.174814 1646 tablet_bootstrap.cc:492] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:46.175380 1646 ts_tablet_manager.cc:1397] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.188s user 0.012s sys 0.008s
I20260808 01:09:46.180727 1645 raft_consensus.cc:348] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.181622 1645 raft_consensus.cc:374] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:46.181929 1645 raft_consensus.cc:729] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:46.182608 1645 consensus_queue.cc:260] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.185091 1645 ts_tablet_manager.cc:1428] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260808 01:09:46.186376 1645 tablet_bootstrap.cc:492] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:46.165083 2085 raft_consensus.cc:2781] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:46.196002 2085 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:46.198051 1646 raft_consensus.cc:348] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.199047 1646 raft_consensus.cc:374] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:46.199344 1646 raft_consensus.cc:729] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:46.200258 1646 consensus_queue.cc:260] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.203119 1646 ts_tablet_manager.cc:1428] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260808 01:09:46.197710 2085 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:46.225039 1944 consensus_queue.cc:1035] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:46.197331 1439 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
W20260808 01:09:46.270031 2134 log.cc:924] Time spent T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
I20260808 01:09:46.262532 2128 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.285629 2128 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:46.307750 1646 tablet_bootstrap.cc:492] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:46.306978 2070 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:46.314565 1648 tablet_bootstrap.cc:492] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:46.315091 1648 ts_tablet_manager.cc:1397] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.343s user 0.030s sys 0.054s
I20260808 01:09:46.323568 1443 raft_consensus.cc:2379] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 1: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:09:46.338846 1648 raft_consensus.cc:348] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.339325 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:46.339730 1648 raft_consensus.cc:374] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:46.341629 1648 raft_consensus.cc:729] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:46.342665 1648 consensus_queue.cc:260] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.340190 1522 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:46.333066 2077 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:46.353615 1522 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:46.353518 2077 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.314484 2070 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.356725 1648 ts_tablet_manager.cc:1428] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.041s user 0.007s sys 0.002s
I20260808 01:09:46.369339 2077 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:46.360165 1410 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:46.381053 2059 raft_consensus.cc:2781] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:46.381794 2059 raft_consensus.cc:686] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:46.385932 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:46.388209 1645 tablet_bootstrap.cc:654] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:46.389829 1646 tablet_bootstrap.cc:654] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:46.373283 2085 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.393728 1340 catalog_manager.cc:5495] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:46.396678 2085 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:09:46.361528 2136 log.cc:924] Time spent T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20260808 01:09:46.403810 1648 tablet_bootstrap.cc:492] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5: Bootstrap starting.
I20260808 01:09:46.406481 2070 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:46.415359 1604 raft_consensus.cc:2379] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 1: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:09:46.417305 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:46.383585 2059 consensus_queue.cc:237] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:09:46.429575 1410 leader_election.cc:341] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:09:46.430454 1390 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:09:46.430300 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:46.431002 1522 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 0.
I20260808 01:09:46.433214 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:46.418737 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:09:46.441613 1568 leader_election.cc:341] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:46.439915 1520 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 0.
I20260808 01:09:46.446302 1567 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:09:46.452605 2085 raft_consensus.cc:2781] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:46.453182 2085 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:46.453672 2085 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:46.457000 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:46.458599 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:46.464999 1525 raft_consensus.cc:2379] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 1: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:09:46.469432 1567 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:46.471216 2131 raft_consensus.cc:2726] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:09:46.481487 1410 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:46.482153 1648 tablet_bootstrap.cc:654] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:46.489837 2059 raft_consensus.cc:2781] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:46.494481 2059 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:46.494858 2059 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:46.504400 1645 tablet_bootstrap.cc:492] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:46.504963 1645 ts_tablet_manager.cc:1397] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.319s user 0.017s sys 0.023s
I20260808 01:09:46.507771 1645 raft_consensus.cc:348] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.508582 1645 raft_consensus.cc:374] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:46.508852 1645 raft_consensus.cc:729] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:46.510720 1646 tablet_bootstrap.cc:492] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:46.511195 1646 ts_tablet_manager.cc:1397] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.209s user 0.019s sys 0.007s
I20260808 01:09:46.511081 1645 consensus_queue.cc:260] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.513749 1645 ts_tablet_manager.cc:1428] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260808 01:09:46.514390 1646 raft_consensus.cc:348] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.515002 1645 tablet_bootstrap.cc:492] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:46.515259 1646 raft_consensus.cc:374] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:46.507725 2085 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.518415 1875 consensus_queue.cc:1035] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:46.520124 2059 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.520612 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:46.521651 1439 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:46.521842 2085 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:46.532125 1645 tablet_bootstrap.cc:654] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:46.535063 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:46.536062 1525 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:46.537022 1439 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:46.539104 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:46.540149 1521 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:46.515615 1646 raft_consensus.cc:729] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:46.542701 2059 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:46.543025 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:09:46.545459 1410 leader_election.cc:341] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:46.542397 1646 consensus_queue.cc:260] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.558974 1646 ts_tablet_manager.cc:1428] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.047s user 0.004s sys 0.004s
I20260808 01:09:46.560225 1646 tablet_bootstrap.cc:492] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78: Bootstrap starting.
I20260808 01:09:46.574157 1648 tablet_bootstrap.cc:492] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5: No bootstrap required, opened a new log
I20260808 01:09:46.574782 1648 ts_tablet_manager.cc:1397] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent bootstrapping tablet: real 0.171s user 0.010s sys 0.013s
I20260808 01:09:46.588093 1568 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:46.591616 2085 raft_consensus.cc:2781] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:46.592379 2085 raft_consensus.cc:686] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:46.592741 1648 raft_consensus.cc:348] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.594051 2085 consensus_queue.cc:237] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.594815 1340 catalog_manager.cc:5495] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:46.594625 1521 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:09:46.598935 1648 raft_consensus.cc:374] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:46.614634 1648 raft_consensus.cc:729] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Initialized, Role: FOLLOWER
I20260808 01:09:46.615437 1648 consensus_queue.cc:260] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.615984 1410 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:46.611102 1525 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:46.617303 2081 raft_consensus.cc:2781] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:46.623163 2106 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:46.624012 2106 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.629281 1648 ts_tablet_manager.cc:1428] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent starting tablet: real 0.054s user 0.004s sys 0.004s
I20260808 01:09:46.643543 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:46.643823 2106 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:46.644763 1599 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 0.
I20260808 01:09:46.646219 1489 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:46.647140 1646 tablet_bootstrap.cc:654] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:46.652765 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:46.653498 1645 tablet_bootstrap.cc:492] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:46.653985 1645 ts_tablet_manager.cc:1397] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.139s user 0.006s sys 0.034s
I20260808 01:09:46.653016 2106 raft_consensus.cc:2781] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:09:46.626406 2081 raft_consensus.cc:686] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:46.661662 2106 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:09:46.662511 1490 leader_election.cc:341] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:09:46.668906 1645 raft_consensus.cc:348] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.670764 1645 raft_consensus.cc:374] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:46.671334 1645 raft_consensus.cc:729] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:46.676154 1645 consensus_queue.cc:260] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.668114 2106 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:46.681372 1645 ts_tablet_manager.cc:1428] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.027s user 0.003s sys 0.005s
I20260808 01:09:46.682346 1645 tablet_bootstrap.cc:492] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d: Bootstrap starting.
I20260808 01:09:46.681880 2081 consensus_queue.cc:237] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.683503 1339 catalog_manager.cc:5495] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:46.708161 1646 tablet_bootstrap.cc:492] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78: No bootstrap required, opened a new log
I20260808 01:09:46.708820 1646 ts_tablet_manager.cc:1397] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78: Time spent bootstrapping tablet: real 0.149s user 0.010s sys 0.013s
I20260808 01:09:46.701505 1875 consensus_queue.cc:1035] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260808 01:09:46.718259 1646 raft_consensus.cc:348] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.733546 1646 raft_consensus.cc:374] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:46.733939 1646 raft_consensus.cc:729] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Initialized, Role: FOLLOWER
I20260808 01:09:46.735087 1646 consensus_queue.cc:260] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.738101 1645 tablet_bootstrap.cc:654] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d: Neither blocks nor log segments found. Creating new log.
I20260808 01:09:46.744191 1339 catalog_manager.cc:5495] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:46.745820 1646 ts_tablet_manager.cc:1428] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78: Time spent starting tablet: real 0.037s user 0.005s sys 0.004s
I20260808 01:09:46.759006 2106 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.789304 1645 tablet_bootstrap.cc:492] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d: No bootstrap required, opened a new log
I20260808 01:09:46.789834 1645 ts_tablet_manager.cc:1397] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent bootstrapping tablet: real 0.108s user 0.015s sys 0.002s
I20260808 01:09:46.798708 2106 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:46.799777 1645 raft_consensus.cc:348] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.800794 1645 raft_consensus.cc:374] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:09:46.801100 1645 raft_consensus.cc:729] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Initialized, Role: FOLLOWER
I20260808 01:09:46.801891 1645 consensus_queue.cc:260] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.804064 1645 ts_tablet_manager.cc:1428] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent starting tablet: real 0.014s user 0.007s sys 0.002s
I20260808 01:09:46.805509 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:46.806336 1596 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:46.807583 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:46.808552 1440 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:46.813719 2081 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:46.814410 2081 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.817385 1596 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:46.817534 2081 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:46.821920 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:46.823876 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:46.829444 1440 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:46.831440 1489 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:46.834160 2078 raft_consensus.cc:2781] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:09:46.834677 2078 raft_consensus.cc:686] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:46.835848 2078 consensus_queue.cc:237] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.846653 1410 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:46.852177 2081 raft_consensus.cc:2726] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:46.852238 1875 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:46.858361 1875 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:46.866027 2128 consensus_queue.cc:1035] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:46.867147 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:46.869686 1875 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:46.874845 1340 catalog_manager.cc:5495] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:46.880303 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:46.886106 1567 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:46.888669 1875 raft_consensus.cc:2726] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:09:46.920521 1439 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:46.926637 2128 consensus_queue.cc:1035] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:46.953636 1599 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:46.999516 32593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 01:09:47.013943 2085 consensus_queue.cc:1035] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260808 01:09:47.024446 2155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:09:47.031730 2156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:09:47.060026 1982 consensus_queue.cc:1035] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260808 01:09:47.108323 2160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:09:47.110841 32593 server_base.cc:1034] running on GCE node
I20260808 01:09:47.114425 32593 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:09:47.114858 32593 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:09:47.115099 32593 hybrid_clock.cc:648] HybridClock initialized: now 1786151387115083 us; error 0 us; skew 500 ppm
I20260808 01:09:47.115923 32593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:09:47.124609 2046 consensus_queue.cc:1035] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:47.125741 32593 webserver.cc:460] Webserver started at http://127.31.212.68:39447/ using document root <none> and password file <none>
I20260808 01:09:47.126608 32593 fs_manager.cc:359] Metadata directory not provided
I20260808 01:09:47.126871 32593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:09:47.127326 32593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:09:47.129627 32593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-3-root/instance:
uuid: "95b661b03f1d460b9657e61d4b0247cd"
format_stamp: "Formatted at 2026-08-08 01:09:47 on dist-test-slave-mvvj"
I20260808 01:09:47.145901 1599 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:09:47.149775 32593 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.012s sys 0.000s
I20260808 01:09:47.188138 2167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:47.189683 2046 consensus_queue.cc:1035] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:47.203971 32593 fs_manager.cc:725] Time spent opening block manager: real 0.050s user 0.003s sys 0.025s
I20260808 01:09:47.215878 32593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-3-root
uuid: "95b661b03f1d460b9657e61d4b0247cd"
format_stamp: "Formatted at 2026-08-08 01:09:47 on dist-test-slave-mvvj"
I20260808 01:09:47.216367 32593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:09:47.272598 2078 consensus_queue.cc:1035] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:09:47.285590 2078 consensus_queue.cc:1035] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:09:47.323307 32593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 01:09:47.325448 32593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:09:47.342141 32593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 01:09:47.376523 32593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:09:47.376967 32593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:47.377563 32593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 01:09:47.377837 32593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:09:47.662194 32593 rpc_server.cc:307] RPC server started. Bound to: 127.31.212.68:34571
I20260808 01:09:47.663048 2233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.212.68:34571 every 8 connection(s)
I20260808 01:09:47.738379 32593 auto_leader_rebalancer-test.cc:413] add a tserver: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:09:47.760474 2234 heartbeater.cc:347] Connected to a master server at 127.31.212.126:33415
I20260808 01:09:47.760937 2234 heartbeater.cc:464] Registering TS with master...
I20260808 01:09:47.761744 2234 heartbeater.cc:511] Master 127.31.212.126:33415 requested a full tablet report, sending...
I20260808 01:09:47.763886 1340 ts_manager.cc:194] Registered new tserver with Master: 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:09:47.765897 1340 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37234
I20260808 01:09:48.769613 2234 heartbeater.cc:503] Master 127.31.212.126:33415 was elected leader, sending a full tablet report...
I20260808 01:09:48.774992 1604 consensus_queue.cc:237] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:48.787194 1521 raft_consensus.cc:1261] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:48.788098 1440 raft_consensus.cc:1261] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:48.789130 1982 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:48.790270 2238 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:48.818255 2238 raft_consensus.cc:2932] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:48.823432 1525 raft_consensus.cc:2932] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:48.823673 1440 raft_consensus.cc:2932] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
W20260808 01:09:48.855532 1565 consensus_peers.cc:487] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44f40a08ace944a88ad5bf8ba22f2a40. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:48.860055 1339 catalog_manager.cc:5495] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:48.897090 1521 consensus_queue.cc:237] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:48.933526 1599 raft_consensus.cc:1261] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:48.935009 2078 consensus_queue.cc:1035] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:48.960412 1445 raft_consensus.cc:1261] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:48.977999 2255 raft_consensus.cc:2932] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
W20260808 01:09:48.981104 1487 consensus_peers.cc:487] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 417bc7d1279d48e89c3599b26117aaf9. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:48.988111 1604 raft_consensus.cc:2932] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:48.996541 1944 consensus_queue.cc:1035] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:49.001360 1521 consensus_queue.cc:237] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:49.017967 1342 catalog_manager.cc:5495] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:49.051252 1596 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:49.052466 1439 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 01:09:49.054051 1487 consensus_peers.cc:487] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e59ebd6ca6845789ae50d410b699ea7. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:49.057734 2255 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:49.056718 1445 raft_consensus.cc:2932] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:49.059742 2260 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:49.089699 2259 raft_consensus.cc:2932] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:49.100334 1604 raft_consensus.cc:2932] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:49.134912 1439 raft_consensus.cc:2932] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:49.155318 1339 catalog_manager.cc:5495] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
W20260808 01:09:49.274055 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 67687.4us
I20260808 01:09:49.298235 2268 ts_tablet_manager.cc:927] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:49.310078 2268 tablet_copy_client.cc:316] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.67:35675
I20260808 01:09:49.336488 1445 consensus_queue.cc:237] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:49.364530 1599 raft_consensus.cc:1261] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:49.366824 1525 raft_consensus.cc:1261] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:49.372722 2128 consensus_queue.cc:1035] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260808 01:09:49.374943 2267 consensus_queue.cc:1035] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:49.471951 1614 tablet_copy_service.cc:140] P a4862d0938c84fc4bd840e4c17acae78: Received BeginTabletCopySession request for tablet 44f40a08ace944a88ad5bf8ba22f2a40 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:34182)
I20260808 01:09:49.473064 1614 tablet_copy_service.cc:161] P a4862d0938c84fc4bd840e4c17acae78: Beginning new tablet copy session on tablet 44f40a08ace944a88ad5bf8ba22f2a40 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:34182: session id = 95b661b03f1d460b9657e61d4b0247cd-44f40a08ace944a88ad5bf8ba22f2a40
I20260808 01:09:49.565093 1614 tablet_copy_source_session.cc:205] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:09:49.567502 2274 ts_tablet_manager.cc:927] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:49.581323 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f40a08ace944a88ad5bf8ba22f2a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:49.594813 2278 ts_tablet_manager.cc:927] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:09:49.631007 2277 log.cc:924] Time spent T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.058s user 0.000s sys 0.002s
I20260808 01:09:49.620261 2274 tablet_copy_client.cc:316] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.66:37093
W20260808 01:09:49.634253 1408 consensus_peers.cc:487] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b316a05b1f7448158ae76530ee0ee127. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:49.664746 1599 raft_consensus.cc:2932] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:49.660841 2247 raft_consensus.cc:2932] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:49.695807 1520 raft_consensus.cc:2932] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:49.744223 2278 tablet_copy_client.cc:316] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.66:37093
I20260808 01:09:49.765043 1339 catalog_manager.cc:5495] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:49.789289 2268 tablet_copy_client.cc:791] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:09:49.790285 2268 tablet_copy_client.cc:655] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:09:49.797704 2268 tablet_copy_client.cc:531] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:09:49.808394 2268 tablet_bootstrap.cc:492] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:09:49.827517 1535 tablet_copy_service.cc:140] P 6b09dbefd24b4d52b385c4dff5697dc5: Received BeginTabletCopySession request for tablet 417bc7d1279d48e89c3599b26117aaf9 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:36560)
I20260808 01:09:49.828028 1534 tablet_copy_service.cc:140] P 6b09dbefd24b4d52b385c4dff5697dc5: Received BeginTabletCopySession request for tablet 1e59ebd6ca6845789ae50d410b699ea7 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:36560)
I20260808 01:09:49.828356 1535 tablet_copy_service.cc:161] P 6b09dbefd24b4d52b385c4dff5697dc5: Beginning new tablet copy session on tablet 417bc7d1279d48e89c3599b26117aaf9 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:36560: session id = 95b661b03f1d460b9657e61d4b0247cd-417bc7d1279d48e89c3599b26117aaf9
I20260808 01:09:49.829118 1534 tablet_copy_service.cc:161] P 6b09dbefd24b4d52b385c4dff5697dc5: Beginning new tablet copy session on tablet 1e59ebd6ca6845789ae50d410b699ea7 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:36560: session id = 95b661b03f1d460b9657e61d4b0247cd-1e59ebd6ca6845789ae50d410b699ea7
I20260808 01:09:49.839710 2288 ts_tablet_manager.cc:927] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
I20260808 01:09:49.840653 1535 tablet_copy_source_session.cc:205] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:09:49.844781 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417bc7d1279d48e89c3599b26117aaf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:49.851512 2288 tablet_copy_client.cc:316] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.65:38291
I20260808 01:09:49.852120 1534 tablet_copy_source_session.cc:205] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:09:49.857908 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e59ebd6ca6845789ae50d410b699ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:49.915540 2278 tablet_copy_client.cc:791] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:09:49.916199 2278 tablet_copy_client.cc:655] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:09:49.925797 2268 tablet_bootstrap.cc:492] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:09:49.930589 2278 tablet_copy_client.cc:531] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:09:49.943464 2278 tablet_bootstrap.cc:492] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:09:49.944912 2268 tablet_bootstrap.cc:492] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:09:49.946285 2268 ts_tablet_manager.cc:1397] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.138s user 0.022s sys 0.012s
I20260808 01:09:49.949898 2268 raft_consensus.cc:348] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:49.955690 1458 tablet_copy_service.cc:140] P 3fb1a6380d27401a831cfe5f86a4cc9d: Received BeginTabletCopySession request for tablet b316a05b1f7448158ae76530ee0ee127 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:60392)
I20260808 01:09:49.980019 2268 raft_consensus.cc:729] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:09:49.980271 1458 tablet_copy_service.cc:161] P 3fb1a6380d27401a831cfe5f86a4cc9d: Beginning new tablet copy session on tablet b316a05b1f7448158ae76530ee0ee127 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:60392: session id = 95b661b03f1d460b9657e61d4b0247cd-b316a05b1f7448158ae76530ee0ee127
I20260808 01:09:49.980876 2268 consensus_queue.cc:260] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:49.970335 2274 tablet_copy_client.cc:791] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:09:49.982733 2274 tablet_copy_client.cc:655] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:09:49.988755 2274 tablet_copy_client.cc:531] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:09:49.976768 2278 tablet_bootstrap.cc:492] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:09:49.993026 2268 ts_tablet_manager.cc:1428] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.046s user 0.007s sys 0.002s
I20260808 01:09:49.997633 2278 tablet_bootstrap.cc:492] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:09:49.998538 2278 ts_tablet_manager.cc:1397] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.055s user 0.024s sys 0.001s
I20260808 01:09:50.012832 2278 raft_consensus.cc:348] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:50.013845 2278 raft_consensus.cc:729] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:09:50.014564 2278 consensus_queue.cc:260] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:50.017166 2278 ts_tablet_manager.cc:1428] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.010s user 0.002s sys 0.003s
I20260808 01:09:50.018419 1614 tablet_copy_service.cc:342] P a4862d0938c84fc4bd840e4c17acae78: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-44f40a08ace944a88ad5bf8ba22f2a40 received from {username='slave'} at 127.0.0.1:34182
I20260808 01:09:50.018985 1614 tablet_copy_service.cc:433] P a4862d0938c84fc4bd840e4c17acae78: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-44f40a08ace944a88ad5bf8ba22f2a40 on tablet 44f40a08ace944a88ad5bf8ba22f2a40 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:09:50.019407 1458 tablet_copy_source_session.cc:205] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:09:50.023375 2288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b316a05b1f7448158ae76530ee0ee127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:50.044215 2274 tablet_bootstrap.cc:492] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:09:50.048171 1534 tablet_copy_service.cc:342] P 6b09dbefd24b4d52b385c4dff5697dc5: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-1e59ebd6ca6845789ae50d410b699ea7 received from {username='slave'} at 127.0.0.1:36560
I20260808 01:09:50.048722 1534 tablet_copy_service.cc:433] P 6b09dbefd24b4d52b385c4dff5697dc5: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-1e59ebd6ca6845789ae50d410b699ea7 on tablet 1e59ebd6ca6845789ae50d410b699ea7 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:09:50.070871 2288 tablet_copy_client.cc:791] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:09:50.071709 2288 tablet_copy_client.cc:655] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:09:50.076227 2288 tablet_copy_client.cc:531] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:09:50.087054 2288 tablet_bootstrap.cc:492] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:09:50.099570 2274 tablet_bootstrap.cc:492] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:09:50.103226 2274 tablet_bootstrap.cc:492] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:09:50.104910 2274 ts_tablet_manager.cc:1397] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.061s user 0.017s sys 0.013s
I20260808 01:09:50.107900 2274 raft_consensus.cc:348] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:50.109089 2274 raft_consensus.cc:729] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:09:50.110914 2274 consensus_queue.cc:260] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:50.113541 2274 ts_tablet_manager.cc:1428] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260808 01:09:50.115319 1534 tablet_copy_service.cc:342] P 6b09dbefd24b4d52b385c4dff5697dc5: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-417bc7d1279d48e89c3599b26117aaf9 received from {username='slave'} at 127.0.0.1:36560
I20260808 01:09:50.115712 1534 tablet_copy_service.cc:433] P 6b09dbefd24b4d52b385c4dff5697dc5: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-417bc7d1279d48e89c3599b26117aaf9 on tablet 417bc7d1279d48e89c3599b26117aaf9 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:09:50.149456 2288 tablet_bootstrap.cc:492] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:09:50.151021 2288 tablet_bootstrap.cc:492] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:09:50.151800 2288 ts_tablet_manager.cc:1397] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.065s user 0.037s sys 0.000s
I20260808 01:09:50.154594 2288 raft_consensus.cc:348] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:50.155565 2288 raft_consensus.cc:729] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:09:50.156317 2288 consensus_queue.cc:260] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:50.161641 2288 ts_tablet_manager.cc:1428] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20260808 01:09:50.163436 1458 tablet_copy_service.cc:342] P 3fb1a6380d27401a831cfe5f86a4cc9d: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-b316a05b1f7448158ae76530ee0ee127 received from {username='slave'} at 127.0.0.1:60392
I20260808 01:09:50.163872 1458 tablet_copy_service.cc:433] P 3fb1a6380d27401a831cfe5f86a4cc9d: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-b316a05b1f7448158ae76530ee0ee127 on tablet b316a05b1f7448158ae76530ee0ee127 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:09:50.326061 2209 raft_consensus.cc:1203] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:09:50.340420 2209 raft_consensus.cc:1203] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:09:50.397481 2209 raft_consensus.cc:1203] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:09:50.524919 2209 raft_consensus.cc:1203] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:09:50.819265 2280 raft_consensus.cc:1050] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:09:50.823316 2280 consensus_queue.cc:237] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:50.834498 1599 raft_consensus.cc:1261] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:50.836086 2275 consensus_queue.cc:1035] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:50.837785 2209 raft_consensus.cc:1261] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:50.840574 2128 consensus_queue.cc:1035] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:50.848524 1525 raft_consensus.cc:1261] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:50.853286 2280 consensus_queue.cc:1035] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:50.876682 2280 raft_consensus.cc:2932] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:50.880776 2209 raft_consensus.cc:2932] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:50.885126 2297 raft_consensus.cc:1050] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:09:50.890969 2297 consensus_queue.cc:237] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:50.903530 2256 raft_consensus.cc:1050] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:09:50.906482 2256 consensus_queue.cc:237] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:50.910012 1342 catalog_manager.cc:5495] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:50.916495 1522 raft_consensus.cc:2932] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:50.951001 1599 raft_consensus.cc:2932] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:50.955600 2209 raft_consensus.cc:1261] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:50.956763 1525 raft_consensus.cc:1261] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:50.957684 2297 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:50.959296 1440 raft_consensus.cc:1261] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:50.959596 2282 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:50.968300 2297 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:51.015920 2209 raft_consensus.cc:1261] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:51.015437 2282 raft_consensus.cc:2932] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.018072 2078 consensus_queue.cc:1035] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:51.029592 1445 raft_consensus.cc:1261] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:51.033756 2256 consensus_queue.cc:1035] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:51.035545 1599 raft_consensus.cc:1261] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:51.069247 1522 raft_consensus.cc:2932] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.087594 2259 consensus_queue.cc:1035] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:51.095754 1445 raft_consensus.cc:2932] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.112087 2078 raft_consensus.cc:1050] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:09:51.115175 2078 consensus_queue.cc:237] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:51.152748 2209 raft_consensus.cc:2932] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.169229 1445 consensus_queue.cc:237] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:51.184087 1341 catalog_manager.cc:5495] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.203794 2322 raft_consensus.cc:2932] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.206245 1597 consensus_queue.cc:237] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:51.210789 2209 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:51.212014 2208 raft_consensus.cc:1261] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:09:51.213665 2128 consensus_queue.cc:1035] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:09:51.214304 2256 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:51.220023 1597 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:51.219516 1596 raft_consensus.cc:2932] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.223237 2324 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:51.231405 1440 raft_consensus.cc:1261] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:09:51.235348 2207 raft_consensus.cc:1261] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:09:51.237198 2280 raft_consensus.cc:2932] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.241447 2282 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:09:51.244369 2207 raft_consensus.cc:2932] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.248236 2313 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:09:51.237265 1445 raft_consensus.cc:2932] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.258701 1594 raft_consensus.cc:1261] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:09:51.271081 2318 raft_consensus.cc:2932] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.278374 1448 raft_consensus.cc:2932] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.309607 2209 raft_consensus.cc:2932] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.316653 1440 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:51.329037 1339 catalog_manager.cc:5495] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.343961 1511 tablet_service.cc:1514] Processing DeleteTablet for tablet b316a05b1f7448158ae76530ee0ee127 with delete_type TABLET_DATA_TOMBSTONED (TS 6b09dbefd24b4d52b385c4dff5697dc5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58250
I20260808 01:09:51.345955 1341 catalog_manager.cc:5495] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.351210 2078 raft_consensus.cc:2932] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.353116 2208 raft_consensus.cc:2932] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.357486 1329 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b316a05b1f7448158ae76530ee0ee127 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:09:51.361032 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 44f40a08ace944a88ad5bf8ba22f2a40 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:09:51.369602 1340 catalog_manager.cc:5495] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.376771 2207 raft_consensus.cc:2932] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.391085 2322 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260808 01:09:51.410667 1340 catalog_manager.cc:5495] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.408759 2128 consensus_queue.cc:1035] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260808 01:09:51.425719 1521 consensus_queue.cc:237] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:51.427428 2327 tablet_replica.cc:330] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5: stopping tablet replica
I20260808 01:09:51.428732 2327 raft_consensus.cc:2229] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:51.432142 2327 raft_consensus.cc:2258] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:51.441227 1594 raft_consensus.cc:2932] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.435218 1511 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f40a08ace944a88ad5bf8ba22f2a40 with delete_type TABLET_DATA_TOMBSTONED (TS 6b09dbefd24b4d52b385c4dff5697dc5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58250
I20260808 01:09:51.452966 1598 raft_consensus.cc:1261] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:09:51.457799 2319 consensus_queue.cc:1035] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260808 01:09:51.460986 2208 raft_consensus.cc:1261] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:09:51.467304 2327 ts_tablet_manager.cc:1905] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:09:51.478816 2259 consensus_queue.cc:1035] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:09:51.489475 2260 raft_consensus.cc:2932] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:09:51.496469 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 63316us
I20260808 01:09:51.495330 2327 ts_tablet_manager.cc:1918] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:09:51.498277 2327 log.cc:1196] T b316a05b1f7448158ae76530ee0ee127 P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-1-root/wals/b316a05b1f7448158ae76530ee0ee127
I20260808 01:09:51.498350 2208 raft_consensus.cc:2932] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.499836 2327 tablet_replica.cc:330] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5: stopping tablet replica
I20260808 01:09:51.499857 1328 catalog_manager.cc:4841] TS 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): tablet b316a05b1f7448158ae76530ee0ee127 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:09:51.500900 2327 raft_consensus.cc:2229] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:51.501551 2327 raft_consensus.cc:2258] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:51.504871 2327 ts_tablet_manager.cc:1905] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:09:51.515470 1594 raft_consensus.cc:2932] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.523811 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 417bc7d1279d48e89c3599b26117aaf9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:09:51.524032 1598 raft_consensus.cc:2932] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.523007 1340 catalog_manager.cc:5495] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.558645 1429 tablet_service.cc:1514] Processing DeleteTablet for tablet 417bc7d1279d48e89c3599b26117aaf9 with delete_type TABLET_DATA_TOMBSTONED (TS 3fb1a6380d27401a831cfe5f86a4cc9d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55036
I20260808 01:09:51.569018 1445 raft_consensus.cc:2932] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.586110 2331 tablet_replica.cc:330] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d: stopping tablet replica
I20260808 01:09:51.587821 2331 raft_consensus.cc:2229] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:51.588688 2331 raft_consensus.cc:2258] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:51.592836 2331 ts_tablet_manager.cc:1905] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:09:51.605873 2327 ts_tablet_manager.cc:1918] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:09:51.606405 2327 log.cc:1196] T 44f40a08ace944a88ad5bf8ba22f2a40 P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-1-root/wals/44f40a08ace944a88ad5bf8ba22f2a40
I20260808 01:09:51.609018 1328 catalog_manager.cc:4841] TS 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): tablet 44f40a08ace944a88ad5bf8ba22f2a40 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:09:51.617648 2331 ts_tablet_manager.cc:1918] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:09:51.618101 2331 log.cc:1196] T 417bc7d1279d48e89c3599b26117aaf9 P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root/wals/417bc7d1279d48e89c3599b26117aaf9
I20260808 01:09:51.619717 1329 catalog_manager.cc:4841] TS 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): tablet 417bc7d1279d48e89c3599b26117aaf9 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:09:51.857468 1522 consensus_queue.cc:237] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:51.863129 2208 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:09:51.865132 2260 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:09:51.874209 2256 raft_consensus.cc:2932] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.880698 1439 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:09:51.884236 2259 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:09:51.886637 2208 raft_consensus.cc:2932] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.900772 1444 raft_consensus.cc:2932] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.905701 1339 catalog_manager.cc:5495] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:51.908066 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 1e59ebd6ca6845789ae50d410b699ea7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:09:51.925328 1586 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e59ebd6ca6845789ae50d410b699ea7 with delete_type TABLET_DATA_TOMBSTONED (TS a4862d0938c84fc4bd840e4c17acae78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45612
I20260808 01:09:51.929960 2332 tablet_replica.cc:330] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78: stopping tablet replica
I20260808 01:09:51.931283 2332 raft_consensus.cc:2229] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:51.932193 2332 raft_consensus.cc:2258] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:51.940479 2332 ts_tablet_manager.cc:1905] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:09:51.977007 2332 ts_tablet_manager.cc:1918] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:09:51.977449 2332 log.cc:1196] T 1e59ebd6ca6845789ae50d410b699ea7 P a4862d0938c84fc4bd840e4c17acae78: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-2-root/wals/1e59ebd6ca6845789ae50d410b699ea7
I20260808 01:09:51.979383 1328 catalog_manager.cc:4841] TS a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): tablet 1e59ebd6ca6845789ae50d410b699ea7 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:09:53.012892 1521 consensus_queue.cc:237] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:09:53.023023 1487 consensus_peers.cc:487] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec7eb407c2934b7f9db6fa7a3ea0fae5. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:53.023370 1594 raft_consensus.cc:1261] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:53.025251 2256 consensus_queue.cc:1035] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:53.026726 1444 raft_consensus.cc:1261] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:53.028258 2256 consensus_queue.cc:1035] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:53.045712 2259 raft_consensus.cc:2932] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:53.051534 1594 raft_consensus.cc:2932] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:53.070705 1521 consensus_queue.cc:237] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:53.081946 1439 raft_consensus.cc:2932] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:53.088868 1340 catalog_manager.cc:5495] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260808 01:09:53.110181 1487 consensus_peers.cc:487] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43bc87d4fdb1436f875364cc78ec3fca. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:53.112140 1598 raft_consensus.cc:1261] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:53.113888 2256 consensus_queue.cc:1035] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:53.124437 1445 raft_consensus.cc:1261] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:53.130237 2259 consensus_queue.cc:1035] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:53.136080 2335 raft_consensus.cc:2932] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:53.145821 1598 raft_consensus.cc:2932] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:53.158645 1439 raft_consensus.cc:2932] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:53.160776 1445 consensus_queue.cc:237] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:09:53.177088 1408 consensus_peers.cc:487] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 260b61a7a7f84a32bda47e79a38ac469. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:53.178133 1594 raft_consensus.cc:1261] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:53.182590 1525 raft_consensus.cc:1261] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:53.184358 2333 consensus_queue.cc:1035] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:53.185591 2280 consensus_queue.cc:1035] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:53.212308 2328 raft_consensus.cc:2932] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:53.205202 1341 catalog_manager.cc:5495] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:53.215039 1521 raft_consensus.cc:2932] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:53.238449 1445 consensus_queue.cc:237] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:53.248204 1594 raft_consensus.cc:2932] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
W20260808 01:09:53.253857 1408 consensus_peers.cc:487] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31f1af2e7c504c919e34f12663452f2a. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:53.254312 1598 raft_consensus.cc:1261] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:53.256603 1339 catalog_manager.cc:5495] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:53.260852 2333 consensus_queue.cc:1035] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:53.261230 1521 raft_consensus.cc:1261] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:53.272035 2333 consensus_queue.cc:1035] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:53.294509 2333 raft_consensus.cc:2932] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:53.306174 1594 raft_consensus.cc:2932] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:53.316197 1521 raft_consensus.cc:2932] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:53.321676 1341 catalog_manager.cc:5495] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:53.444180 2355 ts_tablet_manager.cc:927] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:53.460247 2355 tablet_copy_client.cc:316] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.66:37093
I20260808 01:09:53.466837 1534 tablet_copy_service.cc:140] P 6b09dbefd24b4d52b385c4dff5697dc5: Received BeginTabletCopySession request for tablet ec7eb407c2934b7f9db6fa7a3ea0fae5 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:36560)
I20260808 01:09:53.467412 1534 tablet_copy_service.cc:161] P 6b09dbefd24b4d52b385c4dff5697dc5: Beginning new tablet copy session on tablet ec7eb407c2934b7f9db6fa7a3ea0fae5 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:36560: session id = 95b661b03f1d460b9657e61d4b0247cd-ec7eb407c2934b7f9db6fa7a3ea0fae5
I20260808 01:09:53.479295 1534 tablet_copy_source_session.cc:205] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:09:53.482461 2355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7eb407c2934b7f9db6fa7a3ea0fae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:53.503921 2355 tablet_copy_client.cc:791] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:09:53.504571 2355 tablet_copy_client.cc:655] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:09:53.510988 2355 tablet_copy_client.cc:531] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:09:53.520174 2355 tablet_bootstrap.cc:492] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:09:53.546947 2355 tablet_bootstrap.cc:492] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:09:53.548059 2355 tablet_bootstrap.cc:492] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:09:53.548758 2355 ts_tablet_manager.cc:1397] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.003s
I20260808 01:09:53.551666 2355 raft_consensus.cc:348] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:53.552657 2355 raft_consensus.cc:729] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:09:53.553388 2355 consensus_queue.cc:260] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:53.559159 2355 ts_tablet_manager.cc:1428] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260808 01:09:53.560734 1534 tablet_copy_service.cc:342] P 6b09dbefd24b4d52b385c4dff5697dc5: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-ec7eb407c2934b7f9db6fa7a3ea0fae5 received from {username='slave'} at 127.0.0.1:36560
I20260808 01:09:53.561151 1534 tablet_copy_service.cc:433] P 6b09dbefd24b4d52b385c4dff5697dc5: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-ec7eb407c2934b7f9db6fa7a3ea0fae5 on tablet ec7eb407c2934b7f9db6fa7a3ea0fae5 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:09:53.586225 2355 ts_tablet_manager.cc:927] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:53.592324 2355 tablet_copy_client.cc:316] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.66:37093
I20260808 01:09:53.594039 1534 tablet_copy_service.cc:140] P 6b09dbefd24b4d52b385c4dff5697dc5: Received BeginTabletCopySession request for tablet 43bc87d4fdb1436f875364cc78ec3fca from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:36560)
I20260808 01:09:53.594614 1534 tablet_copy_service.cc:161] P 6b09dbefd24b4d52b385c4dff5697dc5: Beginning new tablet copy session on tablet 43bc87d4fdb1436f875364cc78ec3fca from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:36560: session id = 95b661b03f1d460b9657e61d4b0247cd-43bc87d4fdb1436f875364cc78ec3fca
I20260808 01:09:53.602406 1534 tablet_copy_source_session.cc:205] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:09:53.605350 2355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43bc87d4fdb1436f875364cc78ec3fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:53.622553 2355 tablet_copy_client.cc:791] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:09:53.623273 2355 tablet_copy_client.cc:655] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:09:53.627518 2355 tablet_copy_client.cc:531] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:09:53.636642 2355 tablet_bootstrap.cc:492] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:09:53.648201 2360 ts_tablet_manager.cc:927] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
I20260808 01:09:53.682567 2355 tablet_bootstrap.cc:492] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:09:53.683518 2355 tablet_bootstrap.cc:492] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:09:53.684310 2355 ts_tablet_manager.cc:1397] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.048s user 0.028s sys 0.008s
I20260808 01:09:53.691802 2355 raft_consensus.cc:348] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:53.697487 2355 raft_consensus.cc:729] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:09:53.698302 2355 consensus_queue.cc:260] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:53.709538 2355 ts_tablet_manager.cc:1428] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260808 01:09:53.712023 1534 tablet_copy_service.cc:342] P 6b09dbefd24b4d52b385c4dff5697dc5: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-43bc87d4fdb1436f875364cc78ec3fca received from {username='slave'} at 127.0.0.1:36560
I20260808 01:09:53.712636 1534 tablet_copy_service.cc:433] P 6b09dbefd24b4d52b385c4dff5697dc5: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-43bc87d4fdb1436f875364cc78ec3fca on tablet 43bc87d4fdb1436f875364cc78ec3fca with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:09:53.879141 2360 tablet_copy_client.cc:316] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.65:38291
I20260808 01:09:54.649750 2208 raft_consensus.cc:1203] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:09:54.792683 2259 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:09:54.796913 2259 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.801302 2259 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:54.872328 2372 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:54.873109 2372 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:54.873859 2370 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:09:54.874547 2370 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.875257 2372 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:54.876039 2335 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:09:54.844172 2363 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:54.877010 2259 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:09:54.872212 2380 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:09:54.877595 2335 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.901580 2380 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.903009 2335 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:09:54.880812 1488 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.959s user 0.001s sys 0.032s
I20260808 01:09:54.882411 2259 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.883162 2371 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:54.886193 2364 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:09:54.910534 2371 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.911319 2364 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.912705 2371 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:54.913375 2364 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:54.888226 2363 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:54.915333 2363 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:54.890407 2377 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:09:54.905431 1488 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.983s user 0.002s sys 0.032s
I20260808 01:09:54.916002 2377 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:54.918623 2377 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:54.910003 2259 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:54.864049 2345 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:54.931104 2383 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:54.931255 2373 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:54.931444 2370 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:54.903687 2380 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:54.932281 2208 raft_consensus.cc:1203] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:09:54.932045 2373 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.931771 2383 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.947542 1341 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37234 (request call id 26) took 1242 ms. Trace:
I20260808 01:09:54.948071 1341 rpcz_store.cc:276] 0808 01:09:53.704520 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:53.704735 (+ 215us) service_pool.cc:227] Handling call
0808 01:09:54.946472 (+1241737us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260808 01:09:54.951702 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:54.952657 2373 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:54.955384 2383 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:54.955566 2371 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:54.955729 2335 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:54.939559 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:54.961776 2377 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:09:54.961941 2383 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:54.962450 2377 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:54.962682 2383 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.964624 2377 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:54.964807 2383 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:54.964905 2372 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:54.965673 2372 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.956251 2335 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.966115 2385 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:54.968282 2372 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:54.956198 2371 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:54.968765 2382 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:54.969200 2335 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:54.970042 2371 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:54.933446 2345 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.966935 2385 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.970427 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:54.971218 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:54.968788 2372 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:54.971881 1594 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:54.972035 2372 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.972754 2345 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:54.978091 2372 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:54.969287 2382 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.980772 2382 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:54.982846 2385 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:54.976817 2383 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:54.984431 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:54.985155 1599 raft_consensus.cc:2454] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:54.985322 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:54.986032 1594 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:54.986035 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:54.987419 2345 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:54.987498 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:54.987247 2383 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.992130 2345 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:54.994349 2345 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:54.994836 2383 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.000298 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.002142 2372 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:54.999480 2382 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:55.002923 2382 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.002835 2372 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.004987 2382 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:55.004997 2372 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:55.015736 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.016856 1445 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:09:55.019389 2367 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:55.020085 2380 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:55.033030 2379 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:55.020031 2367 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.033716 2345 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:09:55.034394 2345 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.035588 2367 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.020718 2380 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.024669 2373 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:09:55.026688 2370 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:09:55.055219 2370 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.033762 2379 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.048789 2372 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:55.049734 2380 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.039304 2366 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:09:55.053725 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.054528 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.063270 2259 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:09:55.064389 2259 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.063699 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.068856 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.058840 2370 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.062647 2382 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:55.066993 2364 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:09:55.071007 2374 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:55.065351 2371 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:55.064519 2345 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.074878 2259 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.061604 2372 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.075457 2375 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:09:55.079221 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 61084.2us
I20260808 01:09:55.079509 2375 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:55.080077 2372 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:55.075601 2382 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.081924 2369 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:55.081754 2371 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.058633 2373 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.081779 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.082502 2369 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.100049 2378 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:55.076198 2380 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:55.100769 2378 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.069514 2366 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.106673 2375 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:09:55.107695 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.116516 2380 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.108497 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:09:55.108911 2379 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.109498 2382 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:55.103574 1458 tablet_copy_service.cc:140] P 3fb1a6380d27401a831cfe5f86a4cc9d: Received BeginTabletCopySession request for tablet 260b61a7a7f84a32bda47e79a38ac469 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:60392)
I20260808 01:09:55.111672 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.112044 2373 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.094116 2385 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:55.115463 2365 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:55.115950 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.086498 2384 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:55.099536 2369 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.093086 2374 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.108160 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.130299 2384 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.083654 2367 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:55.113054 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.121038 2366 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.121448 2380 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.125703 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.123113 1458 tablet_copy_service.cc:161] P 3fb1a6380d27401a831cfe5f86a4cc9d: Beginning new tablet copy session on tablet 260b61a7a7f84a32bda47e79a38ac469 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:60392: session id = 95b661b03f1d460b9657e61d4b0247cd-260b61a7a7f84a32bda47e79a38ac469
I20260808 01:09:55.126245 2390 ts_tablet_manager.cc:927] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
I20260808 01:09:55.146168 2378 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:55.126538 2385 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:55.127475 2365 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.083596 2364 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.138195 2374 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:55.141032 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.128167 2371 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:09:55.158322 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 79510.7us
I20260808 01:09:55.148008 1489 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:55.148998 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.152053 2367 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.162113 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.153885 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.168109 2345 raft_consensus.cc:2781] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:09:55.168380 2384 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:55.168694 2345 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:55.168258 2369 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:55.169162 2345 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:55.169409 2369 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.170361 2364 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.175704 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.180636 2367 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.185809 1568 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:55.187431 2385 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:09:55.171638 2369 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.190099 2385 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:55.190232 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:09:55.191825 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.190603 2385 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.192358 2382 raft_consensus.cc:2781] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:09:55.193389 2382 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:55.193779 2382 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:55.194104 2385 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.194257 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.196108 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.197757 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.198207 1458 tablet_copy_source_session.cc:205] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:09:55.202293 1489 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:55.191322 2365 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.205665 1489 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:55.208041 2364 raft_consensus.cc:2781] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:09:55.208504 2364 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:55.208899 2364 raft_consensus.cc:3037] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:55.212771 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.213508 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.212155 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.214994 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.216100 2363 raft_consensus.cc:2781] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:09:55.217382 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.217561 2363 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:55.217970 2363 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:55.221428 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.227097 2370 raft_consensus.cc:1050] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:09:55.228667 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.229359 2372 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:55.229897 2372 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.231168 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.231662 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.235347 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.236229 1599 raft_consensus.cc:2454] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:09:55.241990 2390 tablet_copy_client.cc:316] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.65:38291
I20260808 01:09:55.242723 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.247632 2372 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:55.260629 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.261466 2370 consensus_queue.cc:237] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:55.262799 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.262005 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.264379 2365 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:55.265010 2365 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:55.266335 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.267529 2365 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.268505 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.269313 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.269569 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.270190 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.270624 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.271816 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.275468 2171 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.276412 2345 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:55.277007 2171 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.278438 2377 raft_consensus.cc:2726] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.279050 2377 raft_consensus.cc:2726] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.279001 2345 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.286343 1568 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:09:55.287307 2372 raft_consensus.cc:2726] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.290505 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.292119 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.293643 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.295207 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.297600 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.300259 2382 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.302439 2382 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:55.303629 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:09:55.304414 1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800234600 after lost signal to thread 32596
W20260808 01:09:55.305482 1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800232e20 after lost signal to thread 1376
W20260808 01:09:55.306703 1626 debug-util.cc:398] Leaking SignalData structure 0x7b08001e7ee0 after lost signal to thread 1473
W20260808 01:09:55.307783 1626 debug-util.cc:398] Leaking SignalData structure 0x7b08001ef420 after lost signal to thread 1551
W20260808 01:09:55.308864 1626 debug-util.cc:398] Leaking SignalData structure 0x7b08000fb3e0 after lost signal to thread 1629
I20260808 01:09:55.309649 1458 tablet_copy_service.cc:140] P 3fb1a6380d27401a831cfe5f86a4cc9d: Received BeginTabletCopySession request for tablet 31f1af2e7c504c919e34f12663452f2a from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:60392)
I20260808 01:09:55.310197 1458 tablet_copy_service.cc:161] P 3fb1a6380d27401a831cfe5f86a4cc9d: Beginning new tablet copy session on tablet 31f1af2e7c504c919e34f12663452f2a from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:60392: session id = 95b661b03f1d460b9657e61d4b0247cd-31f1af2e7c504c919e34f12663452f2a
W20260808 01:09:55.310583 1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800229a00 after lost signal to thread 2233
W20260808 01:09:55.311146 1626 debug-util.cc:398] Leaking SignalData structure 0x7b08001096c0 after lost signal to thread 2355
I20260808 01:09:55.313011 2360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 260b61a7a7f84a32bda47e79a38ac469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:55.313433 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.316232 2364 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.316588 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.317687 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.318094 1458 tablet_copy_source_session.cc:205] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:09:55.318935 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.320402 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.321048 1489 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:55.321584 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f1af2e7c504c919e34f12663452f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:09:55.321730 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.321936 2259 raft_consensus.cc:2781] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:09:55.323042 2259 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:55.324707 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.325480 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.327405 2259 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:55.330051 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.331099 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.331349 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:55.318866 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.313004 2363 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.334755 2364 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.337021 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.340586 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.341131 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.342072 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.343608 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.343891 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.345662 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.345829 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.347007 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.349874 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.351653 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:55.356400 2363 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.357208 2259 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.363569 2259 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:55.367241 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:55.367532 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:55.372226 2360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:49270 (request call id 16) took 1748 ms. Trace:
I20260808 01:09:55.372717 2360 rpcz_store.cc:276] 0808 01:09:53.624175 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:09:53.624335 (+ 160us) service_pool.cc:227] Handling call
0808 01:09:53.648659 (+ 24324us) ts_tablet_manager.cc:928] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
0808 01:09:55.311281 (+1662622us) tablet_copy_client.cc:361] Tablet copy session begun
0808 01:09:55.350116 (+ 38835us) tablet_metadata.cc:641] Metadata flushed
0808 01:09:55.350439 (+ 323us) tablet_copy_client.cc:490] Wrote new tablet metadata
0808 01:09:55.371347 (+ 20908us) tablet_copy_client.cc:411] Wrote new consensus metadata
0808 01:09:55.371648 (+ 301us) tablet_replica.cc:205] Creating consensus instance
0808 01:09:55.372062 (+ 414us) ts_tablet_manager.cc:983] Replica registered.
0808 01:09:55.372133 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":405,"tablet-copy.queue_time_us":23330,"thread_start_us":252343,"threads_started":2}
I20260808 01:09:55.374835 2360 tablet_copy_client.cc:791] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:09:55.375802 2360 tablet_copy_client.cc:655] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:09:55.382750 1567 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:55.386636 2384 raft_consensus.cc:2726] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.388159 2208 raft_consensus.cc:1261] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:55.387660 1567 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:55.390740 2392 raft_consensus.cc:2726] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.391541 2360 tablet_copy_client.cc:531] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:09:55.392001 1567 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:55.392671 2259 consensus_queue.cc:1035] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:55.394822 2371 raft_consensus.cc:2726] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.402333 2360 tablet_bootstrap.cc:492] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:09:55.402469 1567 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:55.403239 2390 tablet_copy_client.cc:791] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:09:55.404157 2371 raft_consensus.cc:2726] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.406091 2390 tablet_copy_client.cc:655] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:09:55.405791 1567 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:55.407512 2371 raft_consensus.cc:2726] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.409075 1567 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:55.410051 2371 raft_consensus.cc:2726] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.411396 1567 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:09:55.411895 1489 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.31.212.126:33415, user_credentials={real_user=slave}} blocked reactor thread for 61434.2us
I20260808 01:09:55.412330 2371 raft_consensus.cc:2726] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.413875 1567 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:55.415153 1599 raft_consensus.cc:1226] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 1. Current term is 2. Ops: []
I20260808 01:09:55.416308 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:55.420492 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:55.421309 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.422289 1599 raft_consensus.cc:1261] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:55.423744 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.423995 2390 tablet_copy_client.cc:531] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:09:55.424149 2392 raft_consensus.cc:2726] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.425585 1567 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:55.426156 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.426450 2392 raft_consensus.cc:2726] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.427747 1567 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:55.435074 2390 tablet_bootstrap.cc:492] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:09:55.438014 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.445590 2392 raft_consensus.cc:2726] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.446996 1567 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:55.447804 2392 raft_consensus.cc:2726] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.448355 1567 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:55.449370 2392 raft_consensus.cc:2726] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.449679 1567 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:09:55.450711 2395 log.cc:924] Time spent T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
I20260808 01:09:55.455823 2360 tablet_bootstrap.cc:492] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:09:55.457129 2360 tablet_bootstrap.cc:492] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:09:55.458078 2360 ts_tablet_manager.cc:1397] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.056s user 0.027s sys 0.012s
I20260808 01:09:55.461169 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.463711 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.467579 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.468202 1567 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:55.469273 1567 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:55.471072 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.472424 2392 raft_consensus.cc:2726] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.472433 2378 raft_consensus.cc:2726] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.473086 2371 raft_consensus.cc:2726] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.480036 2392 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:55.480315 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.480692 2392 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.485824 2392 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:55.490231 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:55.493007 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.496688 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.499631 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.501494 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.501608 2360 raft_consensus.cc:348] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:55.504316 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.505681 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.507122 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.507814 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.508612 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.509872 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.508096 2360 raft_consensus.cc:729] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:09:55.510926 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.512511 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:55.515353 2345 consensus_queue.cc:1046] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:55.518880 2363 raft_consensus.cc:3032] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:09:55.519742 2363 raft_consensus.cc:729] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:55.518553 2345 consensus_queue.cc:1035] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260808 01:09:55.521179 2363 consensus_queue.cc:260] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.522981 2363 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:55.529639 2259 raft_consensus.cc:1050] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:09:55.535719 1410 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:09:55.512259 2360 consensus_queue.cc:260] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:55.551949 2373 raft_consensus.cc:2726] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.540197 2259 consensus_queue.cc:237] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:55.578118 2390 tablet_bootstrap.cc:492] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:09:55.580277 2390 tablet_bootstrap.cc:492] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
W20260808 01:09:55.581522 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 81765.9us
I20260808 01:09:55.580731 2360 ts_tablet_manager.cc:1428] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.122s user 0.005s sys 0.006s
I20260808 01:09:55.589749 2390 ts_tablet_manager.cc:1397] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.155s user 0.028s sys 0.015s
I20260808 01:09:55.591190 1490 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.592571 1458 tablet_copy_service.cc:342] P 3fb1a6380d27401a831cfe5f86a4cc9d: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-260b61a7a7f84a32bda47e79a38ac469 received from {username='slave'} at 127.0.0.1:60392
I20260808 01:09:55.593259 1458 tablet_copy_service.cc:433] P 3fb1a6380d27401a831cfe5f86a4cc9d: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-260b61a7a7f84a32bda47e79a38ac469 on tablet 260b61a7a7f84a32bda47e79a38ac469 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:09:55.593911 2398 raft_consensus.cc:2726] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.596024 1490 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.595580 2390 raft_consensus.cc:348] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:55.597255 2390 raft_consensus.cc:729] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:09:55.598531 2390 consensus_queue.cc:260] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:55.603873 2335 raft_consensus.cc:2726] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.604566 1490 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.608568 2390 ts_tablet_manager.cc:1428] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.018s user 0.012s sys 0.001s
I20260808 01:09:55.610828 1490 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.609831 2363 raft_consensus.cc:2726] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.607427 2388 raft_consensus.cc:2932] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:55.617403 1458 tablet_copy_service.cc:342] P 3fb1a6380d27401a831cfe5f86a4cc9d: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-31f1af2e7c504c919e34f12663452f2a received from {username='slave'} at 127.0.0.1:60392
I20260808 01:09:55.618100 1458 tablet_copy_service.cc:433] P 3fb1a6380d27401a831cfe5f86a4cc9d: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-31f1af2e7c504c919e34f12663452f2a on tablet 31f1af2e7c504c919e34f12663452f2a with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:09:55.620556 1410 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.622946 1598 raft_consensus.cc:2932] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:55.626327 1410 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.627483 2380 raft_consensus.cc:2726] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.641914 2208 raft_consensus.cc:1203] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:09:55.644651 1410 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.645901 1490 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.646154 1410 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.624833 2335 raft_consensus.cc:2726] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.648801 1490 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.647189 2385 raft_consensus.cc:2726] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.650923 2369 raft_consensus.cc:2726] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.647425 2363 raft_consensus.cc:2726] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.651614 2345 raft_consensus.cc:2726] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.648150 2380 raft_consensus.cc:2726] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.654109 1490 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.655246 2398 raft_consensus.cc:2726] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.657207 1410 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.659369 1490 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.666867 2208 raft_consensus.cc:1261] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:55.670166 2335 consensus_queue.cc:1035] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:55.671183 2380 raft_consensus.cc:2726] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.676021 1410 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.677456 2365 raft_consensus.cc:2726] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.677804 1598 raft_consensus.cc:1261] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:55.681494 2398 raft_consensus.cc:2726] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.681100 1340 catalog_manager.cc:5495] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:55.679913 2335 consensus_queue.cc:1035] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:55.693490 1490 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.692432 1410 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.694444 2380 raft_consensus.cc:2726] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.694823 1490 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.694928 2345 raft_consensus.cc:2726] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.695881 1410 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.699950 2398 raft_consensus.cc:2726] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.704052 2380 raft_consensus.cc:2726] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.705739 1410 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.707708 1410 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.708552 2380 raft_consensus.cc:2726] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.719985 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:55.725965 2207 raft_consensus.cc:2932] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:55.732880 1567 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:55.733183 1410 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.737468 2378 raft_consensus.cc:2726] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:09:55.736313 2369 raft_consensus.cc:2726] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.746958 1410 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.748309 1410 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.748935 2373 raft_consensus.cc:2726] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.750149 2373 raft_consensus.cc:2726] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.750011 2380 raft_consensus.cc:2726] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.753180 1410 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.754149 2380 raft_consensus.cc:2726] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.754644 1410 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.756531 2380 raft_consensus.cc:2726] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.756894 1410 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.757694 2380 raft_consensus.cc:2726] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.758899 1410 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.765154 2365 raft_consensus.cc:2726] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.765551 1410 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.766736 2365 raft_consensus.cc:2726] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.768031 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:55.768261 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:55.770748 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:55.772172 1445 raft_consensus.cc:1261] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:55.773562 1439 raft_consensus.cc:1261] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:55.780932 1516 raft_consensus.cc:1226] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 1. Current term is 2. Ops: []
I20260808 01:09:55.782613 1525 raft_consensus.cc:1226] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 1. Current term is 2. Ops: []
I20260808 01:09:55.799394 1521 raft_consensus.cc:1226] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 1. Current term is 2. Ops: []
I20260808 01:09:55.801661 1521 raft_consensus.cc:1226] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 1. Current term is 2. Ops: []
I20260808 01:09:55.805420 1490 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.807559 2388 raft_consensus.cc:2726] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.809222 1490 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.810576 1490 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:55.811090 2388 raft_consensus.cc:2726] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.811450 2363 raft_consensus.cc:2726] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:09:55.814534 2363 consensus_queue.cc:1035] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:55.818259 2398 raft_consensus.cc:2932] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:55.820256 2345 consensus_queue.cc:1035] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260808 01:09:55.824455 2373 consensus_queue.cc:1046] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260808 01:09:55.825654 2385 consensus_queue.cc:1046] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:55.827018 2369 raft_consensus.cc:3032] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:09:55.827646 2369 raft_consensus.cc:729] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:55.822032 2207 raft_consensus.cc:2932] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:55.835918 2373 raft_consensus.cc:3032] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:09:55.836513 2373 raft_consensus.cc:729] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:55.830209 2369 consensus_queue.cc:260] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.843205 2387 consensus_queue.cc:1046] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:55.845013 2369 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:55.845706 2387 raft_consensus.cc:3032] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:09:55.837534 2373 consensus_queue.cc:260] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:55.851066 2373 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:55.846218 2387 raft_consensus.cc:729] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:55.852962 2365 consensus_queue.cc:1046] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260808 01:09:55.855540 2387 consensus_queue.cc:260] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.856600 2365 raft_consensus.cc:3032] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:09:55.857614 2365 raft_consensus.cc:729] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:09:55.860311 2387 raft_consensus.cc:3037] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:55.868161 2365 consensus_queue.cc:260] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:55.898753 1599 raft_consensus.cc:2932] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:55.898849 1440 raft_consensus.cc:2932] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:55.933954 1444 raft_consensus.cc:2932] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:55.938546 2365 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:55.969743 1525 consensus_queue.cc:237] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:55.977511 1339 catalog_manager.cc:5495] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:55.993516 2207 raft_consensus.cc:1261] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:09:55.995651 2363 consensus_queue.cc:1035] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:09:56.001252 1603 raft_consensus.cc:1261] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260808 01:09:56.004544 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 128768us
I20260808 01:09:56.009044 2345 consensus_queue.cc:1035] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:09:56.019701 2207 raft_consensus.cc:2932] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:56.028656 1603 raft_consensus.cc:2932] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:56.010717 2345 raft_consensus.cc:2932] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:56.054195 1342 catalog_manager.cc:5495] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:56.056699 2410 raft_consensus.cc:1050] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:09:56.063464 2410 consensus_queue.cc:237] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:56.068568 1429 tablet_service.cc:1514] Processing DeleteTablet for tablet ec7eb407c2934b7f9db6fa7a3ea0fae5 with delete_type TABLET_DATA_TOMBSTONED (TS 3fb1a6380d27401a831cfe5f86a4cc9d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55036
I20260808 01:09:56.071293 2411 tablet_replica.cc:330] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d: stopping tablet replica
I20260808 01:09:56.072631 2411 raft_consensus.cc:2229] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:56.073545 2411 raft_consensus.cc:2258] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:56.077100 2411 ts_tablet_manager.cc:1905] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:09:56.094751 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ec7eb407c2934b7f9db6fa7a3ea0fae5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:09:56.113046 1603 raft_consensus.cc:1261] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:56.113106 2207 raft_consensus.cc:1261] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:56.114552 2365 consensus_queue.cc:1035] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:56.115419 2367 consensus_queue.cc:1035] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:56.128345 1517 raft_consensus.cc:1261] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:09:56.130545 2367 consensus_queue.cc:1035] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:09:56.140589 2387 raft_consensus.cc:2932] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:56.155404 2207 raft_consensus.cc:2932] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:56.171358 2411 ts_tablet_manager.cc:1918] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:09:56.171891 2411 log.cc:1196] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root/wals/ec7eb407c2934b7f9db6fa7a3ea0fae5
I20260808 01:09:56.179563 1341 catalog_manager.cc:5495] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:09:56.175058 1603 raft_consensus.cc:2932] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:56.187126 1525 raft_consensus.cc:2932] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:56.187664 1329 catalog_manager.cc:4841] TS 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): tablet ec7eb407c2934b7f9db6fa7a3ea0fae5 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:09:56.320525 1443 consensus_queue.cc:237] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:56.328105 2207 raft_consensus.cc:1261] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:09:56.329828 1525 raft_consensus.cc:1261] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:09:56.330163 2369 consensus_queue.cc:1035] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:09:56.331487 2367 consensus_queue.cc:1035] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:09:56.346199 2373 raft_consensus.cc:2932] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:56.348744 2208 raft_consensus.cc:2932] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:56.378588 1525 raft_consensus.cc:2932] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:56.381732 1342 catalog_manager.cc:5495] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:56.401203 1517 consensus_queue.cc:237] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:56.409332 1586 tablet_service.cc:1514] Processing DeleteTablet for tablet 260b61a7a7f84a32bda47e79a38ac469 with delete_type TABLET_DATA_TOMBSTONED (TS a4862d0938c84fc4bd840e4c17acae78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45612
I20260808 01:09:56.421772 1329 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 260b61a7a7f84a32bda47e79a38ac469 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:09:56.436745 2417 tablet_replica.cc:330] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78: stopping tablet replica
I20260808 01:09:56.443039 2208 raft_consensus.cc:1261] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:09:56.445503 2345 consensus_queue.cc:1035] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:09:56.457288 2398 raft_consensus.cc:2932] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:56.462066 1440 raft_consensus.cc:1261] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:09:56.467085 2207 raft_consensus.cc:2932] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:56.489295 2363 consensus_queue.cc:1035] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:09:56.492560 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 43bc87d4fdb1436f875364cc78ec3fca with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:09:56.496816 1339 catalog_manager.cc:5495] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:56.499178 2417 raft_consensus.cc:2229] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:56.503343 2417 raft_consensus.cc:2258] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:56.502128 1444 raft_consensus.cc:2932] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:56.515126 2417 ts_tablet_manager.cc:1905] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:09:56.555043 1586 tablet_service.cc:1514] Processing DeleteTablet for tablet 43bc87d4fdb1436f875364cc78ec3fca with delete_type TABLET_DATA_TOMBSTONED (TS a4862d0938c84fc4bd840e4c17acae78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45612
I20260808 01:09:56.614238 2417 ts_tablet_manager.cc:1918] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:09:56.614979 2417 log.cc:1196] T 260b61a7a7f84a32bda47e79a38ac469 P a4862d0938c84fc4bd840e4c17acae78: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-2-root/wals/260b61a7a7f84a32bda47e79a38ac469
I20260808 01:09:56.617892 1328 catalog_manager.cc:4841] TS a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): tablet 260b61a7a7f84a32bda47e79a38ac469 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:09:56.620996 2417 tablet_replica.cc:330] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78: stopping tablet replica
I20260808 01:09:56.622998 2417 raft_consensus.cc:2229] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:56.624469 2417 raft_consensus.cc:2258] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:56.632918 2417 ts_tablet_manager.cc:1905] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:09:56.658035 2417 ts_tablet_manager.cc:1918] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:09:56.658907 2417 log.cc:1196] T 43bc87d4fdb1436f875364cc78ec3fca P a4862d0938c84fc4bd840e4c17acae78: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-2-root/wals/43bc87d4fdb1436f875364cc78ec3fca
I20260808 01:09:56.664544 1328 catalog_manager.cc:4841] TS a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): tablet 43bc87d4fdb1436f875364cc78ec3fca (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:09:56.693454 2370 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:56.694115 2370 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:56.700313 2370 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:56.700906 2378 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:56.701470 2378 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:56.702128 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:56.703353 1596 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:09:56.704245 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:56.704775 1489 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:56.705062 1440 raft_consensus.cc:2454] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:09:56.706310 1568 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:56.706650 2345 raft_consensus.cc:2781] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:09:56.707104 2345 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:56.707181 2418 raft_consensus.cc:2781] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:09:56.707547 2345 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:56.707731 2418 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:56.708498 2418 raft_consensus.cc:3037] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:56.712421 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:56.713474 1517 raft_consensus.cc:2379] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 2: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:09:56.715022 2378 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:56.719408 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:56.720777 2345 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:56.729152 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:56.729904 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:56.730278 1596 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:56.767421 2345 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:56.775034 2367 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:09:56.776003 2367 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:56.780475 2367 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:56.784746 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:56.785713 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:56.787277 1517 raft_consensus.cc:2354] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3.
I20260808 01:09:56.792171 2389 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:56.793051 2389 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:56.806433 2389 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:56.805361 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:56.808274 1517 raft_consensus.cc:2379] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 2: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:09:56.809623 1596 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:09:56.805533 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:56.811689 1489 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:09:56.812403 2418 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:56.813457 2345 raft_consensus.cc:2781] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:09:56.812531 1443 raft_consensus.cc:2454] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:09:56.826344 2345 raft_consensus.cc:686] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:56.829002 1410 leader_election.cc:398] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093) with higher term. Message: Invalid argument: T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3.
I20260808 01:09:56.829360 1410 leader_election.cc:401] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:09:56.831370 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:56.831909 2373 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 3
I20260808 01:09:56.832211 1568 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:56.834309 2389 raft_consensus.cc:2781] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:09:56.836094 2389 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:56.837319 2389 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:56.835060 2345 consensus_queue.cc:237] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:56.847246 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:56.848794 1517 raft_consensus.cc:2379] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 2: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:09:56.852270 2418 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:56.887753 1339 catalog_manager.cc:5495] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:56.916874 2373 raft_consensus.cc:2726] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093) with higher term. Message: Invalid argument: T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3.
I20260808 01:09:56.919353 2389 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:56.929831 2389 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:56.914508 1443 raft_consensus.cc:2454] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:09:56.930790 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:56.931919 1521 raft_consensus.cc:2379] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 2: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:09:56.932730 1568 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:56.934276 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:56.935557 2419 raft_consensus.cc:2781] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:09:56.936414 2419 raft_consensus.cc:686] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:56.938046 2419 consensus_queue.cc:237] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:56.943630 1444 raft_consensus.cc:2454] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:09:56.945876 1568 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:56.947132 2419 raft_consensus.cc:2781] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:09:56.947886 2419 raft_consensus.cc:686] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:56.949513 2419 consensus_queue.cc:237] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:56.959538 1339 catalog_manager.cc:5495] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 1 to 2, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 2 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:56.983635 1339 catalog_manager.cc:5495] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 1 to 2, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 2 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:57.196806 2389 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:57.197482 2389 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:57.201036 2389 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:57.201305 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:57.201843 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:57.202176 1521 raft_consensus.cc:2379] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 2: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:09:57.202754 1443 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:09:57.203822 1568 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:57.204558 2389 raft_consensus.cc:2781] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:09:57.204955 2389 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:57.205333 2389 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:57.216759 2389 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:57.219882 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:57.222004 1443 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:09:57.225533 2370 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:57.229467 1444 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:09:57.231045 2389 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:57.231503 1568 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:09:57.234570 2389 raft_consensus.cc:2781] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:09:57.235718 2389 raft_consensus.cc:686] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:57.236805 2389 consensus_queue.cc:237] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:57.244544 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:57.245334 1521 raft_consensus.cc:2379] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 2: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:09:57.258071 1596 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:09:57.265911 1342 catalog_manager.cc:5495] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 1 to 2, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 2 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:57.271142 2345 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260808 01:09:57.290015 2419 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:57.290570 2419 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:57.292685 2419 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:57.294138 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:09:57.295011 1517 raft_consensus.cc:2379] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 2: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:09:57.296128 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:57.296790 1443 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:09:57.300450 1568 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:57.302330 2418 raft_consensus.cc:2781] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:09:57.302735 2418 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:09:57.303128 2418 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:09:57.310104 2418 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:57.312784 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:09:57.313642 1517 raft_consensus.cc:2379] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 2: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:09:57.313426 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:57.324644 1443 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:09:57.311784 2418 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:09:57.329499 1568 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:09:57.332824 2418 raft_consensus.cc:2781] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:09:57.333307 2418 raft_consensus.cc:686] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:57.334153 2418 consensus_queue.cc:237] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:57.358785 2335 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:57.359290 2335 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:57.361405 2335 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:57.362398 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:57.363727 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:57.364372 1443 raft_consensus.cc:2454] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:09:57.365619 1490 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:57.366375 2335 raft_consensus.cc:2781] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:09:57.366766 2335 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:57.367111 2335 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:57.370867 1339 catalog_manager.cc:5495] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 1 to 2, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 2 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:09:57.374428 2335 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:57.376536 2335 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:57.378499 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:57.379163 1443 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:57.385854 1443 raft_consensus.cc:2454] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:09:57.388571 1490 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:57.390626 2335 raft_consensus.cc:2781] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:09:57.390353 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:57.392549 2335 raft_consensus.cc:686] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:57.394256 2335 consensus_queue.cc:237] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:57.410537 1339 catalog_manager.cc:5495] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 2 to 3, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:57.421943 1525 raft_consensus.cc:1226] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
I20260808 01:09:57.423862 2378 consensus_queue.cc:1046] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:57.425088 2378 consensus_queue.cc:1225] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer 6b09dbefd24b4d52b385c4dff5697dc5 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260808 01:09:57.425659 2378 raft_consensus.cc:3032] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:09:57.425969 2378 raft_consensus.cc:729] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:57.426770 2378 consensus_queue.cc:260] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:57.428844 2378 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:57.521905 1440 raft_consensus.cc:1261] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:57.523253 2418 consensus_queue.cc:1035] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:57.535270 1525 raft_consensus.cc:1261] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:09:57.540773 2419 consensus_queue.cc:1035] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:57.652068 2335 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:09:57.652590 2335 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:57.654659 2335 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:57.655227 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:09:57.656008 1440 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:09:57.656801 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:09:57.657054 1490 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:09:57.657991 2335 raft_consensus.cc:2781] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:09:57.658388 2335 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:09:57.658805 2335 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:57.665696 2335 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:57.668051 2335 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:09:57.668756 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:09:57.669283 1440 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:57.670599 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:09:57.675457 1440 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:09:57.676513 1490 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:09:57.677297 2335 raft_consensus.cc:2781] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:09:57.678082 2335 raft_consensus.cc:686] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:09:57.680462 2335 consensus_queue.cc:237] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:57.695034 1342 catalog_manager.cc:5495] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 2 to 3, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:09:57.698734 1525 raft_consensus.cc:1261] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:09:57.700928 2418 consensus_queue.cc:1035] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:57.738183 2207 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Advancing to term 2
I20260808 01:09:57.740046 1443 raft_consensus.cc:1261] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:09:57.742754 2418 consensus_queue.cc:1035] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:57.755000 2207 raft_consensus.cc:1261] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:09:57.756608 2418 consensus_queue.cc:1035] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:09:57.824558 1443 raft_consensus.cc:1261] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:09:57.826094 2378 raft_consensus.cc:1050] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:09:57.826510 2345 consensus_queue.cc:1035] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:57.831461 2378 consensus_queue.cc:237] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:57.846462 1444 raft_consensus.cc:1226] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
I20260808 01:09:57.848968 2418 consensus_queue.cc:1046] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:57.851467 2419 raft_consensus.cc:3032] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:09:57.852075 2419 raft_consensus.cc:729] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:09:57.852300 2207 raft_consensus.cc:1261] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:09:57.853255 2419 consensus_queue.cc:260] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:09:57.854980 2389 consensus_queue.cc:1035] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:09:57.863543 1598 raft_consensus.cc:1261] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260808 01:09:57.864071 1598 pending_rounds.cc:76] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78: Aborting all ops after (but not including) 1
I20260808 01:09:57.864322 1598 pending_rounds.cc:97] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260808 01:09:57.864660 1598 raft_consensus.cc:2866] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:09:57.867875 2259 consensus_queue.cc:1035] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:57.854969 2419 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:09:57.924265 1525 raft_consensus.cc:1261] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:09:57.935698 2389 consensus_queue.cc:1035] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260808 01:09:58.044682 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 60954.8us
I20260808 01:09:58.055869 1448 raft_consensus.cc:1261] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:09:58.057690 2389 raft_consensus.cc:2932] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:58.063148 2208 raft_consensus.cc:2932] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:58.127645 1598 raft_consensus.cc:1261] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260808 01:09:58.127996 1598 pending_rounds.cc:76] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78: Aborting all ops after (but not including) 1
I20260808 01:09:58.128197 1598 pending_rounds.cc:97] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260808 01:09:58.128402 1598 raft_consensus.cc:2866] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:09:58.131716 1342 catalog_manager.cc:5495] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 2 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:58.139595 2441 consensus_queue.cc:1035] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:09:58.157347 1445 raft_consensus.cc:2932] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:58.176208 1521 raft_consensus.cc:2932] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:58.192214 2398 consensus_queue.cc:1035] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:58.212920 1445 raft_consensus.cc:1261] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:09:58.216779 2345 consensus_queue.cc:1035] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:58.589562 1603 consensus_queue.cc:237] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:09:58.594578 2208 raft_consensus.cc:1261] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:09:58.596961 1445 raft_consensus.cc:1261] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:09:58.597230 2389 consensus_queue.cc:1035] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:09:58.600708 2441 consensus_queue.cc:1035] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:09:58.605268 2418 raft_consensus.cc:2932] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:58.609639 1448 raft_consensus.cc:2932] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:58.614215 2208 raft_consensus.cc:2932] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:09:58.616207 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 31f1af2e7c504c919e34f12663452f2a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:09:58.626674 1342 catalog_manager.cc:5495] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index 4 to 5, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New cstate: current_term: 2 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:09:58.639981 1511 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f1af2e7c504c919e34f12663452f2a with delete_type TABLET_DATA_TOMBSTONED (TS 6b09dbefd24b4d52b385c4dff5697dc5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58250
I20260808 01:09:58.641721 2453 tablet_replica.cc:330] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5: stopping tablet replica
I20260808 01:09:58.642634 2453 raft_consensus.cc:2229] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:09:58.643421 2453 raft_consensus.cc:2258] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:09:58.647400 2453 ts_tablet_manager.cc:1905] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:09:58.667297 2453 ts_tablet_manager.cc:1918] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260808 01:09:58.667711 2453 log.cc:1196] T 31f1af2e7c504c919e34f12663452f2a P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-1-root/wals/31f1af2e7c504c919e34f12663452f2a
I20260808 01:09:58.669270 1328 catalog_manager.cc:4841] TS 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): tablet 31f1af2e7c504c919e34f12663452f2a (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:09:59.731771 1521 consensus_queue.cc:237] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:59.738775 1603 raft_consensus.cc:1261] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:59.740480 1445 raft_consensus.cc:1261] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:59.740779 2449 consensus_queue.cc:1035] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:59.742316 2259 consensus_queue.cc:1035] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260808 01:09:59.745616 1487 consensus_peers.cc:487] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efd5468b2c6e4509ae17a98f9509ff03. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:59.751060 2449 raft_consensus.cc:2932] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:59.752753 1603 raft_consensus.cc:2932] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:59.758504 1445 raft_consensus.cc:2932] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:59.773288 1521 consensus_queue.cc:237] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:59.786689 1603 raft_consensus.cc:1261] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 01:09:59.787847 1487 consensus_peers.cc:487] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 212d8f7da4b84fae81802fe11cdd4b81. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:59.788899 2460 consensus_queue.cc:1035] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:59.790863 1342 catalog_manager.cc:5495] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:59.794126 1445 raft_consensus.cc:1261] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:59.798846 2460 consensus_queue.cc:1035] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:59.810282 2259 raft_consensus.cc:2932] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:59.812562 1598 raft_consensus.cc:2932] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:59.816928 1445 raft_consensus.cc:2932] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:59.840915 1445 consensus_queue.cc:237] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:59.852475 1339 catalog_manager.cc:5495] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:59.855085 1521 raft_consensus.cc:1261] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:59.855108 1598 raft_consensus.cc:1261] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 01:09:59.855899 1408 consensus_peers.cc:487] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5353a0d85aff48078011ca6efaf9ddfa. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:59.869194 2446 consensus_queue.cc:1035] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:59.870513 2455 consensus_queue.cc:1035] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:59.883071 2444 raft_consensus.cc:2932] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:59.884881 1521 raft_consensus.cc:2932] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:59.885766 1598 raft_consensus.cc:2932] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:59.899865 1521 consensus_queue.cc:237] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:09:59.904536 1342 catalog_manager.cc:5495] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
W20260808 01:09:59.910980 1487 consensus_peers.cc:487] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 002e3bea322d445e8f5f8f09e2442db4. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:09:59.912550 1598 raft_consensus.cc:1261] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:59.914849 1445 raft_consensus.cc:1261] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:09:59.915035 2460 consensus_queue.cc:1035] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:09:59.916373 2449 consensus_queue.cc:1035] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:09:59.927099 2259 raft_consensus.cc:2932] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:59.943427 1598 raft_consensus.cc:2932] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:59.944069 1445 raft_consensus.cc:2932] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:09:59.956501 1342 catalog_manager.cc:5495] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:00.263345 2475 ts_tablet_manager.cc:927] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:00.272094 2475 tablet_copy_client.cc:316] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.66:37093
I20260808 01:10:00.273937 1534 tablet_copy_service.cc:140] P 6b09dbefd24b4d52b385c4dff5697dc5: Received BeginTabletCopySession request for tablet efd5468b2c6e4509ae17a98f9509ff03 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:36560)
I20260808 01:10:00.274575 1534 tablet_copy_service.cc:161] P 6b09dbefd24b4d52b385c4dff5697dc5: Beginning new tablet copy session on tablet efd5468b2c6e4509ae17a98f9509ff03 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:36560: session id = 95b661b03f1d460b9657e61d4b0247cd-efd5468b2c6e4509ae17a98f9509ff03
I20260808 01:10:00.283349 1534 tablet_copy_source_session.cc:205] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:00.287592 2475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efd5468b2c6e4509ae17a98f9509ff03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:00.306042 2475 tablet_copy_client.cc:791] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:00.306809 2475 tablet_copy_client.cc:655] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:00.311062 2475 tablet_copy_client.cc:531] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:00.315876 2477 ts_tablet_manager.cc:927] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:00.318466 2477 tablet_copy_client.cc:316] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.66:37093
I20260808 01:10:00.320434 1534 tablet_copy_service.cc:140] P 6b09dbefd24b4d52b385c4dff5697dc5: Received BeginTabletCopySession request for tablet 002e3bea322d445e8f5f8f09e2442db4 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:36560)
I20260808 01:10:00.321023 1534 tablet_copy_service.cc:161] P 6b09dbefd24b4d52b385c4dff5697dc5: Beginning new tablet copy session on tablet 002e3bea322d445e8f5f8f09e2442db4 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:36560: session id = 95b661b03f1d460b9657e61d4b0247cd-002e3bea322d445e8f5f8f09e2442db4
I20260808 01:10:00.322296 2475 tablet_bootstrap.cc:492] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:00.331254 1534 tablet_copy_source_session.cc:205] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:00.334813 2477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 002e3bea322d445e8f5f8f09e2442db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:00.355147 2475 tablet_bootstrap.cc:492] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:00.356351 2475 tablet_bootstrap.cc:492] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:00.357425 2475 ts_tablet_manager.cc:1397] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.035s user 0.018s sys 0.016s
I20260808 01:10:00.360713 2477 tablet_copy_client.cc:791] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:00.361450 2477 tablet_copy_client.cc:655] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:00.361490 2475 raft_consensus.cc:348] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:00.362706 2475 raft_consensus.cc:729] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:00.363533 2475 consensus_queue.cc:260] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:00.369047 2477 tablet_copy_client.cc:531] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:00.382202 2475 ts_tablet_manager.cc:1428] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.024s user 0.007s sys 0.015s
I20260808 01:10:00.384332 1534 tablet_copy_service.cc:342] P 6b09dbefd24b4d52b385c4dff5697dc5: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-efd5468b2c6e4509ae17a98f9509ff03 received from {username='slave'} at 127.0.0.1:36560
I20260808 01:10:00.384735 1534 tablet_copy_service.cc:433] P 6b09dbefd24b4d52b385c4dff5697dc5: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-efd5468b2c6e4509ae17a98f9509ff03 on tablet efd5468b2c6e4509ae17a98f9509ff03 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:00.392167 2477 tablet_bootstrap.cc:492] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:00.407454 2475 ts_tablet_manager.cc:927] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:00.415822 2477 tablet_bootstrap.cc:492] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:00.417698 2475 tablet_copy_client.cc:316] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.66:37093
I20260808 01:10:00.417850 2477 tablet_bootstrap.cc:492] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:00.418704 2477 ts_tablet_manager.cc:1397] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.027s user 0.021s sys 0.004s
I20260808 01:10:00.419772 1534 tablet_copy_service.cc:140] P 6b09dbefd24b4d52b385c4dff5697dc5: Received BeginTabletCopySession request for tablet 212d8f7da4b84fae81802fe11cdd4b81 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:36560)
I20260808 01:10:00.420356 1534 tablet_copy_service.cc:161] P 6b09dbefd24b4d52b385c4dff5697dc5: Beginning new tablet copy session on tablet 212d8f7da4b84fae81802fe11cdd4b81 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:36560: session id = 95b661b03f1d460b9657e61d4b0247cd-212d8f7da4b84fae81802fe11cdd4b81
I20260808 01:10:00.425995 2477 raft_consensus.cc:348] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:00.426957 2477 raft_consensus.cc:729] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:00.427598 1534 tablet_copy_source_session.cc:205] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:00.427621 2477 consensus_queue.cc:260] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:00.431137 2475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 212d8f7da4b84fae81802fe11cdd4b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:00.431272 2477 ts_tablet_manager.cc:1428] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260808 01:10:00.440209 1534 tablet_copy_service.cc:342] P 6b09dbefd24b4d52b385c4dff5697dc5: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-002e3bea322d445e8f5f8f09e2442db4 received from {username='slave'} at 127.0.0.1:36560
I20260808 01:10:00.440797 1534 tablet_copy_service.cc:433] P 6b09dbefd24b4d52b385c4dff5697dc5: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-002e3bea322d445e8f5f8f09e2442db4 on tablet 002e3bea322d445e8f5f8f09e2442db4 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:00.449357 2477 ts_tablet_manager.cc:927] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
I20260808 01:10:00.451337 2477 tablet_copy_client.cc:316] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.65:38291
I20260808 01:10:00.453536 1458 tablet_copy_service.cc:140] P 3fb1a6380d27401a831cfe5f86a4cc9d: Received BeginTabletCopySession request for tablet 5353a0d85aff48078011ca6efaf9ddfa from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:60392)
I20260808 01:10:00.454092 1458 tablet_copy_service.cc:161] P 3fb1a6380d27401a831cfe5f86a4cc9d: Beginning new tablet copy session on tablet 5353a0d85aff48078011ca6efaf9ddfa from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:60392: session id = 95b661b03f1d460b9657e61d4b0247cd-5353a0d85aff48078011ca6efaf9ddfa
I20260808 01:10:00.456238 2475 tablet_copy_client.cc:791] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:00.456815 2475 tablet_copy_client.cc:655] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:00.464560 1458 tablet_copy_source_session.cc:205] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:00.466092 2475 tablet_copy_client.cc:531] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:00.473239 2477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5353a0d85aff48078011ca6efaf9ddfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:00.478180 2475 tablet_bootstrap.cc:492] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:00.494621 2477 tablet_copy_client.cc:791] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:00.495323 2477 tablet_copy_client.cc:655] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:00.499234 2477 tablet_copy_client.cc:531] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:00.509385 2477 tablet_bootstrap.cc:492] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:00.519616 2475 tablet_bootstrap.cc:492] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:00.520773 2475 tablet_bootstrap.cc:492] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:00.521472 2475 ts_tablet_manager.cc:1397] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.043s user 0.016s sys 0.011s
I20260808 01:10:00.524386 2475 raft_consensus.cc:348] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:00.525285 2475 raft_consensus.cc:729] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:00.525987 2475 consensus_queue.cc:260] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:00.528635 2475 ts_tablet_manager.cc:1428] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 01:10:00.531502 1534 tablet_copy_service.cc:342] P 6b09dbefd24b4d52b385c4dff5697dc5: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-212d8f7da4b84fae81802fe11cdd4b81 received from {username='slave'} at 127.0.0.1:36560
I20260808 01:10:00.531946 1534 tablet_copy_service.cc:433] P 6b09dbefd24b4d52b385c4dff5697dc5: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-212d8f7da4b84fae81802fe11cdd4b81 on tablet 212d8f7da4b84fae81802fe11cdd4b81 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:00.535986 2477 tablet_bootstrap.cc:492] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:00.536995 2477 tablet_bootstrap.cc:492] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:00.537714 2477 ts_tablet_manager.cc:1397] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.000s
I20260808 01:10:00.540551 2477 raft_consensus.cc:348] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:00.542272 2477 raft_consensus.cc:729] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:00.542936 2477 consensus_queue.cc:260] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:00.545919 2477 ts_tablet_manager.cc:1428] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260808 01:10:00.548386 1458 tablet_copy_service.cc:342] P 3fb1a6380d27401a831cfe5f86a4cc9d: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-5353a0d85aff48078011ca6efaf9ddfa received from {username='slave'} at 127.0.0.1:60392
I20260808 01:10:00.548799 1458 tablet_copy_service.cc:433] P 3fb1a6380d27401a831cfe5f86a4cc9d: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-5353a0d85aff48078011ca6efaf9ddfa on tablet 5353a0d85aff48078011ca6efaf9ddfa with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:00.844064 2208 raft_consensus.cc:1203] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:00.865149 2209 raft_consensus.cc:1203] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:00.866020 2208 raft_consensus.cc:1203] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:00.935449 2208 raft_consensus.cc:1203] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:01.242808 2460 raft_consensus.cc:1050] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:01.245651 2460 consensus_queue.cc:237] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:01.251645 2208 raft_consensus.cc:1261] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:01.253368 2460 consensus_queue.cc:1035] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:01.253682 1598 raft_consensus.cc:1261] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:01.253973 1445 raft_consensus.cc:1261] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:01.255563 2490 consensus_queue.cc:1035] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:01.256639 2460 consensus_queue.cc:1035] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:01.267035 2474 raft_consensus.cc:2932] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.268765 1445 raft_consensus.cc:2932] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.285303 1598 raft_consensus.cc:2932] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.288156 2208 raft_consensus.cc:2932] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.304279 1341 catalog_manager.cc:5495] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.327142 2259 raft_consensus.cc:1050] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:01.329901 2259 consensus_queue.cc:237] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:01.346889 2209 raft_consensus.cc:1261] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:01.348413 2490 consensus_queue.cc:1035] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:01.349587 1603 raft_consensus.cc:1261] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:01.351763 1440 raft_consensus.cc:1261] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:01.353470 2490 consensus_queue.cc:1035] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:01.354593 2474 consensus_queue.cc:1035] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:01.360634 2491 raft_consensus.cc:1050] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:01.364218 2491 consensus_queue.cc:237] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:01.390143 2208 raft_consensus.cc:1261] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:01.393793 1598 raft_consensus.cc:1261] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:01.393837 2460 consensus_queue.cc:1035] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:01.396476 2460 consensus_queue.cc:1035] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:01.397579 1445 raft_consensus.cc:1261] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:01.405009 1525 consensus_queue.cc:237] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:01.406281 2460 consensus_queue.cc:1035] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:01.405473 2474 raft_consensus.cc:2932] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.407611 1445 raft_consensus.cc:2932] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.414654 2207 raft_consensus.cc:1261] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:01.416767 1603 raft_consensus.cc:1261] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:01.418370 2490 consensus_queue.cc:1035] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:01.428352 2209 raft_consensus.cc:2932] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.427376 2259 consensus_queue.cc:1035] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20260808 01:10:01.447395 2490 raft_consensus.cc:2932] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.452111 2206 raft_consensus.cc:2932] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.474918 1339 catalog_manager.cc:5495] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:10:01.487324 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 63699.2us
I20260808 01:10:01.500113 2444 raft_consensus.cc:1050] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
W20260808 01:10:01.503158 2505 log.cc:924] Time spent T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.101s user 0.001s sys 0.000s
I20260808 01:10:01.505039 2444 consensus_queue.cc:237] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:01.512820 1596 raft_consensus.cc:2932] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.542286 1339 catalog_manager.cc:5495] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.545292 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 212d8f7da4b84fae81802fe11cdd4b81 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:01.550750 2490 raft_consensus.cc:2932] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.553035 1598 raft_consensus.cc:2932] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.560923 1445 raft_consensus.cc:2932] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.571082 1603 raft_consensus.cc:1261] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:01.572551 2206 raft_consensus.cc:1261] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:01.572732 2446 consensus_queue.cc:1035] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:01.575733 2209 raft_consensus.cc:2932] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.578609 2515 consensus_queue.cc:1035] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:01.546886 1596 raft_consensus.cc:2932] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.592943 1517 raft_consensus.cc:1261] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:01.594380 2515 consensus_queue.cc:1035] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:01.595984 1521 consensus_queue.cc:237] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:01.609042 2209 raft_consensus.cc:1261] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:01.612092 2514 consensus_queue.cc:1035] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260808 01:10:01.631945 1604 raft_consensus.cc:1261] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:01.644726 2511 raft_consensus.cc:2932] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.646144 2206 raft_consensus.cc:2932] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.653368 1429 tablet_service.cc:1514] Processing DeleteTablet for tablet 212d8f7da4b84fae81802fe11cdd4b81 with delete_type TABLET_DATA_TOMBSTONED (TS 3fb1a6380d27401a831cfe5f86a4cc9d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55036
I20260808 01:10:01.651813 1342 catalog_manager.cc:5495] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.660490 2446 raft_consensus.cc:2932] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.659512 2491 consensus_queue.cc:1035] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260808 01:10:01.668319 1604 raft_consensus.cc:2932] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.670161 1598 raft_consensus.cc:2932] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.677546 1516 raft_consensus.cc:2932] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.688581 2207 raft_consensus.cc:2932] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.704366 2520 tablet_replica.cc:330] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d: stopping tablet replica
I20260808 01:10:01.710269 2520 raft_consensus.cc:2229] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:01.710649 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet efd5468b2c6e4509ae17a98f9509ff03 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:01.705235 1525 consensus_queue.cc:237] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:01.713615 2520 raft_consensus.cc:2258] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:01.722685 1444 raft_consensus.cc:1261] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:01.724028 2474 consensus_queue.cc:1035] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:01.728889 1341 catalog_manager.cc:5495] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.731544 2206 raft_consensus.cc:1261] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:01.735013 2511 consensus_queue.cc:1035] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:01.737205 2520 ts_tablet_manager.cc:1905] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:01.737613 1445 raft_consensus.cc:2932] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.749315 1339 catalog_manager.cc:5495] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:01.757256 1444 consensus_queue.cc:237] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:01.744796 2207 raft_consensus.cc:2932] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.743077 2474 raft_consensus.cc:2932] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.763962 1604 raft_consensus.cc:1261] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:01.766485 2206 raft_consensus.cc:1261] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:01.767977 2455 consensus_queue.cc:1035] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:01.770403 2446 consensus_queue.cc:1035] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:01.775748 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 002e3bea322d445e8f5f8f09e2442db4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:01.775962 1429 tablet_service.cc:1514] Processing DeleteTablet for tablet efd5468b2c6e4509ae17a98f9509ff03 with delete_type TABLET_DATA_TOMBSTONED (TS 3fb1a6380d27401a831cfe5f86a4cc9d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55036
I20260808 01:10:01.778052 2515 raft_consensus.cc:2932] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.779331 2206 raft_consensus.cc:2932] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.806231 1329 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 5353a0d85aff48078011ca6efaf9ddfa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:01.795428 1339 catalog_manager.cc:5495] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.807037 1604 raft_consensus.cc:2932] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.819304 1586 tablet_service.cc:1514] Processing DeleteTablet for tablet 002e3bea322d445e8f5f8f09e2442db4 with delete_type TABLET_DATA_TOMBSTONED (TS a4862d0938c84fc4bd840e4c17acae78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45612
I20260808 01:10:01.826299 2520 ts_tablet_manager.cc:1918] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:01.826959 2520 log.cc:1196] T 212d8f7da4b84fae81802fe11cdd4b81 P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root/wals/212d8f7da4b84fae81802fe11cdd4b81
I20260808 01:10:01.829892 2520 tablet_replica.cc:330] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d: stopping tablet replica
I20260808 01:10:01.831305 2520 raft_consensus.cc:2229] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:01.832252 2520 raft_consensus.cc:2258] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:01.828896 1329 catalog_manager.cc:4841] TS 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): tablet 212d8f7da4b84fae81802fe11cdd4b81 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:01.836205 1342 catalog_manager.cc:5495] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:01.839113 2520 ts_tablet_manager.cc:1905] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:01.844416 2521 tablet_replica.cc:330] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78: stopping tablet replica
I20260808 01:10:01.846201 2521 raft_consensus.cc:2229] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:01.847056 2521 raft_consensus.cc:2258] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:01.850342 2521 ts_tablet_manager.cc:1905] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:01.862907 2520 ts_tablet_manager.cc:1918] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:01.863544 2520 log.cc:1196] T efd5468b2c6e4509ae17a98f9509ff03 P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root/wals/efd5468b2c6e4509ae17a98f9509ff03
I20260808 01:10:01.865413 1329 catalog_manager.cc:4841] TS 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): tablet efd5468b2c6e4509ae17a98f9509ff03 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:01.872697 2521 ts_tablet_manager.cc:1918] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:01.873417 2521 log.cc:1196] T 002e3bea322d445e8f5f8f09e2442db4 P a4862d0938c84fc4bd840e4c17acae78: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-2-root/wals/002e3bea322d445e8f5f8f09e2442db4
I20260808 01:10:01.878031 1511 tablet_service.cc:1514] Processing DeleteTablet for tablet 5353a0d85aff48078011ca6efaf9ddfa with delete_type TABLET_DATA_TOMBSTONED (TS 6b09dbefd24b4d52b385c4dff5697dc5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58250
I20260808 01:10:01.880136 2522 tablet_replica.cc:330] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5: stopping tablet replica
I20260808 01:10:01.881335 1328 catalog_manager.cc:4841] TS a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): tablet 002e3bea322d445e8f5f8f09e2442db4 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:01.894521 2522 raft_consensus.cc:2229] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:01.896819 2522 raft_consensus.cc:2258] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:01.901872 2522 ts_tablet_manager.cc:1905] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:01.931905 2522 ts_tablet_manager.cc:1918] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:01.932513 2522 log.cc:1196] T 5353a0d85aff48078011ca6efaf9ddfa P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-1-root/wals/5353a0d85aff48078011ca6efaf9ddfa
I20260808 01:10:01.936017 1328 catalog_manager.cc:4841] TS 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): tablet 5353a0d85aff48078011ca6efaf9ddfa (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:02.877269 1445 consensus_queue.cc:237] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:10:02.888206 1408 consensus_peers.cc:487] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ad0577278e046ad8387388b9e5e98a0. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:02.889114 1603 raft_consensus.cc:1261] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:02.890692 1525 raft_consensus.cc:1261] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:02.892760 2515 consensus_queue.cc:1035] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:02.893836 2455 consensus_queue.cc:1035] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:02.904012 2527 raft_consensus.cc:2932] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:02.905787 1603 raft_consensus.cc:2932] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:02.914060 1525 raft_consensus.cc:2932] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:02.918515 1516 consensus_queue.cc:237] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:02.929139 1339 catalog_manager.cc:5495] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
W20260808 01:10:02.970902 1487 consensus_peers.cc:487] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 921f3eb5150b4105a4b68355cca8cb7a. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:02.973271 1445 raft_consensus.cc:1261] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:02.975279 2474 consensus_queue.cc:1035] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:02.975332 1604 raft_consensus.cc:1261] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:02.978745 2474 consensus_queue.cc:1035] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:02.989436 2531 raft_consensus.cc:2932] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:02.991590 1604 raft_consensus.cc:2932] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:02.999953 1444 raft_consensus.cc:2932] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:03.015239 1342 catalog_manager.cc:5495] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:03.016204 1516 consensus_queue.cc:237] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:10:03.038290 1487 consensus_peers.cc:487] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eba23837e3184c0d99703d8b5b0accbe. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:03.038280 1604 raft_consensus.cc:1261] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:03.039885 1444 raft_consensus.cc:1261] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:03.039912 2474 consensus_queue.cc:1035] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:03.041302 2531 consensus_queue.cc:1035] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:03.048292 2474 raft_consensus.cc:2932] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:03.049891 1604 raft_consensus.cc:2932] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:03.050858 1444 raft_consensus.cc:2932] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:03.062752 1516 consensus_queue.cc:237] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:03.071786 1342 catalog_manager.cc:5495] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
W20260808 01:10:03.080106 1487 consensus_peers.cc:487] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc54c3f2f5e04f63b88243c0d2d6babe. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:03.082404 1604 raft_consensus.cc:1261] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:03.089525 2474 consensus_queue.cc:1035] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:03.099576 1444 raft_consensus.cc:1261] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:03.101023 2539 consensus_queue.cc:1035] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:03.102429 2531 raft_consensus.cc:2932] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:03.104367 1604 raft_consensus.cc:2932] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:03.119212 1444 raft_consensus.cc:2932] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:03.123039 1342 catalog_manager.cc:5495] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:03.399478 2544 ts_tablet_manager.cc:927] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
I20260808 01:10:03.401738 2544 tablet_copy_client.cc:316] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.65:38291
I20260808 01:10:03.403326 1458 tablet_copy_service.cc:140] P 3fb1a6380d27401a831cfe5f86a4cc9d: Received BeginTabletCopySession request for tablet 3ad0577278e046ad8387388b9e5e98a0 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:60392)
I20260808 01:10:03.404002 1458 tablet_copy_service.cc:161] P 3fb1a6380d27401a831cfe5f86a4cc9d: Beginning new tablet copy session on tablet 3ad0577278e046ad8387388b9e5e98a0 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:60392: session id = 95b661b03f1d460b9657e61d4b0247cd-3ad0577278e046ad8387388b9e5e98a0
I20260808 01:10:03.417292 1458 tablet_copy_source_session.cc:205] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:03.420969 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad0577278e046ad8387388b9e5e98a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:03.441780 2544 tablet_copy_client.cc:791] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:03.442817 2544 tablet_copy_client.cc:655] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:03.447274 2544 tablet_copy_client.cc:531] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:03.448046 2546 ts_tablet_manager.cc:927] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:03.450320 2546 tablet_copy_client.cc:316] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.66:37093
I20260808 01:10:03.452188 1534 tablet_copy_service.cc:140] P 6b09dbefd24b4d52b385c4dff5697dc5: Received BeginTabletCopySession request for tablet 921f3eb5150b4105a4b68355cca8cb7a from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:36560)
I20260808 01:10:03.452781 1534 tablet_copy_service.cc:161] P 6b09dbefd24b4d52b385c4dff5697dc5: Beginning new tablet copy session on tablet 921f3eb5150b4105a4b68355cca8cb7a from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:36560: session id = 95b661b03f1d460b9657e61d4b0247cd-921f3eb5150b4105a4b68355cca8cb7a
I20260808 01:10:03.459277 2544 tablet_bootstrap.cc:492] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:03.468103 1534 tablet_copy_source_session.cc:205] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:03.474058 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921f3eb5150b4105a4b68355cca8cb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:03.498220 2546 tablet_copy_client.cc:791] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:03.498849 2546 tablet_copy_client.cc:655] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:03.504765 2546 tablet_copy_client.cc:531] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:03.527369 2546 tablet_bootstrap.cc:492] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:03.529559 2544 tablet_bootstrap.cc:492] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:03.530683 2544 tablet_bootstrap.cc:492] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:03.531468 2544 ts_tablet_manager.cc:1397] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.072s user 0.035s sys 0.018s
I20260808 01:10:03.534435 2544 raft_consensus.cc:348] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:03.535409 2544 raft_consensus.cc:729] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:03.536100 2544 consensus_queue.cc:260] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:03.540068 2544 ts_tablet_manager.cc:1428] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.008s user 0.008s sys 0.004s
I20260808 01:10:03.541965 1458 tablet_copy_service.cc:342] P 3fb1a6380d27401a831cfe5f86a4cc9d: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-3ad0577278e046ad8387388b9e5e98a0 received from {username='slave'} at 127.0.0.1:60392
I20260808 01:10:03.542382 1458 tablet_copy_service.cc:433] P 3fb1a6380d27401a831cfe5f86a4cc9d: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-3ad0577278e046ad8387388b9e5e98a0 on tablet 3ad0577278e046ad8387388b9e5e98a0 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:03.552136 2544 ts_tablet_manager.cc:927] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:03.563149 2544 tablet_copy_client.cc:316] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.66:37093
I20260808 01:10:03.564864 1534 tablet_copy_service.cc:140] P 6b09dbefd24b4d52b385c4dff5697dc5: Received BeginTabletCopySession request for tablet eba23837e3184c0d99703d8b5b0accbe from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:36560)
I20260808 01:10:03.565399 1534 tablet_copy_service.cc:161] P 6b09dbefd24b4d52b385c4dff5697dc5: Beginning new tablet copy session on tablet eba23837e3184c0d99703d8b5b0accbe from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:36560: session id = 95b661b03f1d460b9657e61d4b0247cd-eba23837e3184c0d99703d8b5b0accbe
I20260808 01:10:03.565969 2546 tablet_bootstrap.cc:492] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:03.566962 2546 tablet_bootstrap.cc:492] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:03.567698 2546 ts_tablet_manager.cc:1397] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.041s user 0.019s sys 0.017s
I20260808 01:10:03.570163 2546 raft_consensus.cc:348] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:03.571240 2546 raft_consensus.cc:729] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:03.571976 2546 consensus_queue.cc:260] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:03.574126 2546 ts_tablet_manager.cc:1428] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260808 01:10:03.574792 1534 tablet_copy_source_session.cc:205] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:03.581722 1534 tablet_copy_service.cc:342] P 6b09dbefd24b4d52b385c4dff5697dc5: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-921f3eb5150b4105a4b68355cca8cb7a received from {username='slave'} at 127.0.0.1:36560
I20260808 01:10:03.582392 1534 tablet_copy_service.cc:433] P 6b09dbefd24b4d52b385c4dff5697dc5: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-921f3eb5150b4105a4b68355cca8cb7a on tablet 921f3eb5150b4105a4b68355cca8cb7a with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:03.582719 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba23837e3184c0d99703d8b5b0accbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:03.603544 2544 tablet_copy_client.cc:791] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:03.604351 2544 tablet_copy_client.cc:655] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:03.607890 2544 tablet_copy_client.cc:531] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:03.617062 2544 tablet_bootstrap.cc:492] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:03.626308 2546 ts_tablet_manager.cc:927] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:03.631016 2546 tablet_copy_client.cc:316] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.66:37093
I20260808 01:10:03.635226 1534 tablet_copy_service.cc:140] P 6b09dbefd24b4d52b385c4dff5697dc5: Received BeginTabletCopySession request for tablet bc54c3f2f5e04f63b88243c0d2d6babe from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:36560)
I20260808 01:10:03.635779 1534 tablet_copy_service.cc:161] P 6b09dbefd24b4d52b385c4dff5697dc5: Beginning new tablet copy session on tablet bc54c3f2f5e04f63b88243c0d2d6babe from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:36560: session id = 95b661b03f1d460b9657e61d4b0247cd-bc54c3f2f5e04f63b88243c0d2d6babe
I20260808 01:10:03.645783 2544 tablet_bootstrap.cc:492] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:03.646821 2544 tablet_bootstrap.cc:492] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:03.647500 2544 ts_tablet_manager.cc:1397] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.008s
I20260808 01:10:03.649140 1534 tablet_copy_source_session.cc:205] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:03.650491 2544 raft_consensus.cc:348] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:03.651288 2544 raft_consensus.cc:729] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:03.652098 2544 consensus_queue.cc:260] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:03.652566 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc54c3f2f5e04f63b88243c0d2d6babe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:03.656921 2544 ts_tablet_manager.cc:1428] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.009s user 0.011s sys 0.000s
I20260808 01:10:03.659755 1534 tablet_copy_service.cc:342] P 6b09dbefd24b4d52b385c4dff5697dc5: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-eba23837e3184c0d99703d8b5b0accbe received from {username='slave'} at 127.0.0.1:36560
I20260808 01:10:03.660952 1534 tablet_copy_service.cc:433] P 6b09dbefd24b4d52b385c4dff5697dc5: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-eba23837e3184c0d99703d8b5b0accbe on tablet eba23837e3184c0d99703d8b5b0accbe with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:03.669453 2546 tablet_copy_client.cc:791] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:03.670117 2546 tablet_copy_client.cc:655] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:03.675390 2546 tablet_copy_client.cc:531] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:03.685818 2546 tablet_bootstrap.cc:492] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:03.705919 2546 tablet_bootstrap.cc:492] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:03.706816 2546 tablet_bootstrap.cc:492] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:03.707490 2546 ts_tablet_manager.cc:1397] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.022s user 0.019s sys 0.000s
I20260808 01:10:03.710129 2546 raft_consensus.cc:348] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:03.710757 2546 raft_consensus.cc:729] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:03.711349 2546 consensus_queue.cc:260] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:03.713565 2546 ts_tablet_manager.cc:1428] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260808 01:10:03.716012 1534 tablet_copy_service.cc:342] P 6b09dbefd24b4d52b385c4dff5697dc5: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-bc54c3f2f5e04f63b88243c0d2d6babe received from {username='slave'} at 127.0.0.1:36560
I20260808 01:10:03.716400 1534 tablet_copy_service.cc:433] P 6b09dbefd24b4d52b385c4dff5697dc5: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-bc54c3f2f5e04f63b88243c0d2d6babe on tablet bc54c3f2f5e04f63b88243c0d2d6babe with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:03.875216 2206 raft_consensus.cc:1203] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:03.938091 2206 raft_consensus.cc:1203] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:04.053745 2206 raft_consensus.cc:1203] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:04.118207 2206 raft_consensus.cc:1203] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20260808 01:10:04.295485 2474 raft_consensus.cc:1050] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:04.298158 2474 consensus_queue.cc:237] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:04.306661 2206 raft_consensus.cc:1261] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:04.307621 1604 raft_consensus.cc:1261] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:04.309602 1443 raft_consensus.cc:1261] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:04.312168 2543 consensus_queue.cc:1035] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:04.314112 2543 consensus_queue.cc:1035] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:04.317742 2531 consensus_queue.cc:1035] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:04.348012 2543 raft_consensus.cc:2932] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.354516 1444 raft_consensus.cc:2932] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.360544 2207 raft_consensus.cc:2932] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.374555 1339 catalog_manager.cc:5495] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.383980 1604 raft_consensus.cc:2932] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.429286 1517 consensus_queue.cc:237] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:04.434731 2207 raft_consensus.cc:1261] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:04.438166 2558 consensus_queue.cc:1035] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:04.439198 1604 raft_consensus.cc:1261] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:04.441200 2558 consensus_queue.cc:1035] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:04.447732 2559 raft_consensus.cc:2932] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.452055 2206 raft_consensus.cc:2932] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.452054 1604 raft_consensus.cc:2932] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.458256 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 921f3eb5150b4105a4b68355cca8cb7a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:04.470166 1340 catalog_manager.cc:5495] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:04.483934 2515 raft_consensus.cc:1050] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:04.486145 1429 tablet_service.cc:1514] Processing DeleteTablet for tablet 921f3eb5150b4105a4b68355cca8cb7a with delete_type TABLET_DATA_TOMBSTONED (TS 3fb1a6380d27401a831cfe5f86a4cc9d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55036
I20260808 01:10:04.486541 2515 consensus_queue.cc:237] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:04.492182 2569 tablet_replica.cc:330] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d: stopping tablet replica
I20260808 01:10:04.493348 2569 raft_consensus.cc:2229] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:04.494040 2569 raft_consensus.cc:2258] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:04.501865 1604 raft_consensus.cc:1261] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:04.502696 2206 raft_consensus.cc:1261] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:04.503767 2527 consensus_queue.cc:1035] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:04.505277 2556 consensus_queue.cc:1035] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:04.512678 1525 raft_consensus.cc:1261] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:04.514475 2556 consensus_queue.cc:1035] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:04.522431 2559 raft_consensus.cc:1050] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:04.522472 2569 ts_tablet_manager.cc:1905] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:04.534336 2556 raft_consensus.cc:2932] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.536269 1517 raft_consensus.cc:2932] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.544080 2543 raft_consensus.cc:1050] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:04.546594 2543 consensus_queue.cc:237] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:04.548147 2207 raft_consensus.cc:2932] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.552058 2559 consensus_queue.cc:237] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:04.558436 1340 catalog_manager.cc:5495] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.565678 2569 ts_tablet_manager.cc:1918] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:04.566143 2569 log.cc:1196] T 921f3eb5150b4105a4b68355cca8cb7a P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root/wals/921f3eb5150b4105a4b68355cca8cb7a
I20260808 01:10:04.576735 1604 raft_consensus.cc:2932] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.588490 1443 raft_consensus.cc:1261] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 01:10:04.593588 2206 raft_consensus.cc:1261] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:04.594525 2577 consensus_queue.cc:1035] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:04.595142 2474 consensus_queue.cc:1035] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:04.601992 1603 raft_consensus.cc:1261] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:04.603343 1598 raft_consensus.cc:1261] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 01:10:04.609730 2474 consensus_queue.cc:1035] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:04.611132 2474 consensus_queue.cc:1035] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:04.582479 2207 raft_consensus.cc:1261] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 01:10:04.621475 2576 consensus_queue.cc:1035] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:04.627609 1440 raft_consensus.cc:1261] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:04.631017 2579 consensus_queue.cc:1035] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:04.635960 1329 catalog_manager.cc:4841] TS 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): tablet 921f3eb5150b4105a4b68355cca8cb7a (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:04.635120 2576 raft_consensus.cc:2932] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.659905 2207 raft_consensus.cc:2932] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.661381 2579 raft_consensus.cc:2932] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.663002 1443 consensus_queue.cc:237] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:04.671619 2206 raft_consensus.cc:1261] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:04.672307 1596 raft_consensus.cc:1261] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:04.673034 2527 consensus_queue.cc:1035] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:04.672061 1440 raft_consensus.cc:2932] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.674896 2527 consensus_queue.cc:1035] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:04.684919 2556 raft_consensus.cc:2932] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.697847 1599 raft_consensus.cc:2932] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.708231 1444 raft_consensus.cc:2932] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.710439 2206 raft_consensus.cc:2932] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.706103 1604 raft_consensus.cc:2932] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.731002 2207 raft_consensus.cc:2932] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.731127 1340 catalog_manager.cc:5495] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.749040 1329 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3ad0577278e046ad8387388b9e5e98a0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:04.762414 1340 catalog_manager.cc:5495] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.782836 1341 catalog_manager.cc:5495] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.799161 1603 raft_consensus.cc:2932] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.826711 1511 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ad0577278e046ad8387388b9e5e98a0 with delete_type TABLET_DATA_TOMBSTONED (TS 6b09dbefd24b4d52b385c4dff5697dc5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58250
I20260808 01:10:04.836102 1517 consensus_queue.cc:237] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:04.838052 1525 consensus_queue.cc:237] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:04.840174 2588 tablet_replica.cc:330] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5: stopping tablet replica
I20260808 01:10:04.853443 2206 raft_consensus.cc:1261] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:04.854020 2588 raft_consensus.cc:2229] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:04.860605 2588 raft_consensus.cc:2258] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:04.876734 2588 ts_tablet_manager.cc:1905] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:04.881470 1445 raft_consensus.cc:1261] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:10:04.864566 2559 consensus_queue.cc:1035] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20260808 01:10:04.889839 2531 consensus_queue.cc:1035] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:04.905797 1444 raft_consensus.cc:1261] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:04.916679 2559 consensus_queue.cc:1035] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:04.926414 2206 raft_consensus.cc:1261] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:10:04.928761 2474 raft_consensus.cc:2932] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.932471 1445 raft_consensus.cc:2932] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.929445 2577 raft_consensus.cc:2932] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.933632 2543 consensus_queue.cc:1035] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:04.925233 2209 raft_consensus.cc:2932] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.937321 1444 raft_consensus.cc:2932] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.964310 2206 raft_consensus.cc:2932] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.968747 1342 catalog_manager.cc:5495] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index 4 to 5, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:04.992770 2588 ts_tablet_manager.cc:1918] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:04.993513 2588 log.cc:1196] T 3ad0577278e046ad8387388b9e5e98a0 P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-1-root/wals/3ad0577278e046ad8387388b9e5e98a0
I20260808 01:10:04.994972 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet eba23837e3184c0d99703d8b5b0accbe with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:05.013998 1341 catalog_manager.cc:5495] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:05.024652 1328 catalog_manager.cc:4841] TS 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): tablet 3ad0577278e046ad8387388b9e5e98a0 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:05.028692 1586 tablet_service.cc:1514] Processing DeleteTablet for tablet bc54c3f2f5e04f63b88243c0d2d6babe with delete_type TABLET_DATA_TOMBSTONED (TS a4862d0938c84fc4bd840e4c17acae78 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45612
I20260808 01:10:05.041137 2592 tablet_replica.cc:330] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78: stopping tablet replica
I20260808 01:10:05.050894 1586 tablet_service.cc:1514] Processing DeleteTablet for tablet eba23837e3184c0d99703d8b5b0accbe with delete_type TABLET_DATA_TOMBSTONED (TS a4862d0938c84fc4bd840e4c17acae78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45612
I20260808 01:10:05.053494 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet bc54c3f2f5e04f63b88243c0d2d6babe with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:10:05.074990 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 213354us
I20260808 01:10:05.081733 2592 raft_consensus.cc:2229] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:05.083141 2592 raft_consensus.cc:2258] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:05.088557 2592 ts_tablet_manager.cc:1905] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:05.160533 2592 ts_tablet_manager.cc:1918] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20260808 01:10:05.161504 2592 log.cc:1196] T bc54c3f2f5e04f63b88243c0d2d6babe P a4862d0938c84fc4bd840e4c17acae78: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-2-root/wals/bc54c3f2f5e04f63b88243c0d2d6babe
I20260808 01:10:05.164026 1328 catalog_manager.cc:4841] TS a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): tablet bc54c3f2f5e04f63b88243c0d2d6babe (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:05.164510 2592 tablet_replica.cc:330] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78: stopping tablet replica
I20260808 01:10:05.166723 2592 raft_consensus.cc:2229] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:05.167791 2592 raft_consensus.cc:2258] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:05.173162 2592 ts_tablet_manager.cc:1905] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:05.200785 2592 ts_tablet_manager.cc:1918] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:05.201458 2592 log.cc:1196] T eba23837e3184c0d99703d8b5b0accbe P a4862d0938c84fc4bd840e4c17acae78: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-2-root/wals/eba23837e3184c0d99703d8b5b0accbe
I20260808 01:10:05.203624 1328 catalog_manager.cc:4841] TS a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): tablet eba23837e3184c0d99703d8b5b0accbe (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:06.146477 1603 consensus_queue.cc:237] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:06.152758 1525 raft_consensus.cc:1261] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:06.153831 1444 raft_consensus.cc:1261] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 01:10:06.154178 1565 consensus_peers.cc:487] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3583c46ad2f4aa8875c06ef46cde18a. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:06.154109 2593 consensus_queue.cc:1035] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:06.155211 2589 consensus_queue.cc:1035] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:06.163172 2589 raft_consensus.cc:2932] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:06.165006 1444 raft_consensus.cc:2932] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:06.174587 1516 raft_consensus.cc:2932] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:06.187361 1445 consensus_queue.cc:237] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:06.188791 1340 catalog_manager.cc:5495] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:06.199616 1603 raft_consensus.cc:1261] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 01:10:06.201572 1408 consensus_peers.cc:487] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58547863eabc48eda1552ed4240f68d3. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:06.202836 2587 consensus_queue.cc:1035] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:06.203719 1516 raft_consensus.cc:1261] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:06.205112 2586 consensus_queue.cc:1035] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:06.213848 2587 raft_consensus.cc:2932] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:06.216395 1603 raft_consensus.cc:2932] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:06.217257 1516 raft_consensus.cc:2932] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:06.237918 1516 consensus_queue.cc:237] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:06.257270 1342 catalog_manager.cc:5495] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:06.267769 1603 raft_consensus.cc:1261] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:06.268744 1445 raft_consensus.cc:1261] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:06.269091 2543 consensus_queue.cc:1035] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:06.270109 2474 consensus_queue.cc:1035] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 01:10:06.271008 1487 consensus_peers.cc:487] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a19c1759594d4722abadf576352f9a98. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:06.281626 1445 raft_consensus.cc:2932] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:06.281855 1603 raft_consensus.cc:2932] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:06.280575 2607 raft_consensus.cc:2932] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:06.299676 1340 catalog_manager.cc:5495] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:06.304417 1603 consensus_queue.cc:237] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:10:06.319159 1565 consensus_peers.cc:487] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ccc57605ae24bf4bacee4e4741bbe95. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:06.322913 1516 raft_consensus.cc:1261] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:06.324357 1444 raft_consensus.cc:1261] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:06.324743 2589 consensus_queue.cc:1035] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260808 01:10:06.328022 2523 consensus_queue.cc:1035] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:06.336148 2523 raft_consensus.cc:2932] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:06.339260 1516 raft_consensus.cc:2932] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:06.367051 1341 catalog_manager.cc:5495] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:06.361490 1445 raft_consensus.cc:2932] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:06.704671 2612 ts_tablet_manager.cc:927] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
I20260808 01:10:06.711452 2612 tablet_copy_client.cc:316] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.65:38291
I20260808 01:10:06.714006 1458 tablet_copy_service.cc:140] P 3fb1a6380d27401a831cfe5f86a4cc9d: Received BeginTabletCopySession request for tablet 58547863eabc48eda1552ed4240f68d3 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:60392)
I20260808 01:10:06.714601 1458 tablet_copy_service.cc:161] P 3fb1a6380d27401a831cfe5f86a4cc9d: Beginning new tablet copy session on tablet 58547863eabc48eda1552ed4240f68d3 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:60392: session id = 95b661b03f1d460b9657e61d4b0247cd-58547863eabc48eda1552ed4240f68d3
I20260808 01:10:06.727450 2614 ts_tablet_manager.cc:927] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:06.733100 1458 tablet_copy_source_session.cc:205] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:06.737787 2618 ts_tablet_manager.cc:927] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:06.738979 2612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58547863eabc48eda1552ed4240f68d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:06.742482 2618 tablet_copy_client.cc:316] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.67:35675
I20260808 01:10:06.745998 1614 tablet_copy_service.cc:140] P a4862d0938c84fc4bd840e4c17acae78: Received BeginTabletCopySession request for tablet b3583c46ad2f4aa8875c06ef46cde18a from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:34182)
I20260808 01:10:06.746593 1614 tablet_copy_service.cc:161] P a4862d0938c84fc4bd840e4c17acae78: Beginning new tablet copy session on tablet b3583c46ad2f4aa8875c06ef46cde18a from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:34182: session id = 95b661b03f1d460b9657e61d4b0247cd-b3583c46ad2f4aa8875c06ef46cde18a
I20260808 01:10:06.747109 2614 tablet_copy_client.cc:316] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.67:35675
I20260808 01:10:06.753301 1613 tablet_copy_service.cc:140] P a4862d0938c84fc4bd840e4c17acae78: Received BeginTabletCopySession request for tablet 1ccc57605ae24bf4bacee4e4741bbe95 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:34182)
I20260808 01:10:06.753971 1613 tablet_copy_service.cc:161] P a4862d0938c84fc4bd840e4c17acae78: Beginning new tablet copy session on tablet 1ccc57605ae24bf4bacee4e4741bbe95 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:34182: session id = 95b661b03f1d460b9657e61d4b0247cd-1ccc57605ae24bf4bacee4e4741bbe95
I20260808 01:10:06.756518 1614 tablet_copy_source_session.cc:205] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:06.760799 2620 ts_tablet_manager.cc:927] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:06.761322 2612 tablet_copy_client.cc:791] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:06.761966 2612 tablet_copy_client.cc:655] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:06.767481 2612 tablet_copy_client.cc:531] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:06.773651 2620 tablet_copy_client.cc:316] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.66:37093
I20260808 01:10:06.775473 1534 tablet_copy_service.cc:140] P 6b09dbefd24b4d52b385c4dff5697dc5: Received BeginTabletCopySession request for tablet a19c1759594d4722abadf576352f9a98 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:36560)
I20260808 01:10:06.776242 1534 tablet_copy_service.cc:161] P 6b09dbefd24b4d52b385c4dff5697dc5: Beginning new tablet copy session on tablet a19c1759594d4722abadf576352f9a98 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:36560: session id = 95b661b03f1d460b9657e61d4b0247cd-a19c1759594d4722abadf576352f9a98
I20260808 01:10:06.778985 1613 tablet_copy_source_session.cc:205] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:06.779489 2618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3583c46ad2f4aa8875c06ef46cde18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:06.783987 2614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccc57605ae24bf4bacee4e4741bbe95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:06.789829 1534 tablet_copy_source_session.cc:205] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:06.801330 2612 tablet_bootstrap.cc:492] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:06.804862 2620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19c1759594d4722abadf576352f9a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:06.825870 2614 tablet_copy_client.cc:791] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:06.826520 2614 tablet_copy_client.cc:655] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:06.833365 2618 tablet_copy_client.cc:791] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:06.834264 2618 tablet_copy_client.cc:655] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:06.840656 2614 tablet_copy_client.cc:531] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:06.845394 2612 tablet_bootstrap.cc:492] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:06.845891 2618 tablet_copy_client.cc:531] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:06.846418 2612 tablet_bootstrap.cc:492] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:06.847332 2612 ts_tablet_manager.cc:1397] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.046s user 0.018s sys 0.020s
I20260808 01:10:06.864559 2614 tablet_bootstrap.cc:492] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:06.865865 2620 tablet_copy_client.cc:791] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:06.866811 2620 tablet_copy_client.cc:655] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:06.858244 2612 raft_consensus.cc:348] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:06.871033 2612 raft_consensus.cc:729] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:06.886631 2612 consensus_queue.cc:260] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:06.890321 2618 tablet_bootstrap.cc:492] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:06.892894 2620 tablet_copy_client.cc:531] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:06.919114 2612 ts_tablet_manager.cc:1428] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.071s user 0.010s sys 0.010s
I20260808 01:10:06.928170 1458 tablet_copy_service.cc:342] P 3fb1a6380d27401a831cfe5f86a4cc9d: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-58547863eabc48eda1552ed4240f68d3 received from {username='slave'} at 127.0.0.1:60392
I20260808 01:10:06.928860 1458 tablet_copy_service.cc:433] P 3fb1a6380d27401a831cfe5f86a4cc9d: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-58547863eabc48eda1552ed4240f68d3 on tablet 58547863eabc48eda1552ed4240f68d3 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:06.944468 2620 tablet_bootstrap.cc:492] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:07.061815 2614 tablet_bootstrap.cc:492] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:07.063868 2614 tablet_bootstrap.cc:492] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:07.065089 2614 ts_tablet_manager.cc:1397] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.201s user 0.040s sys 0.041s
I20260808 01:10:07.096632 2618 tablet_bootstrap.cc:492] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:07.098757 2618 tablet_bootstrap.cc:492] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:07.100906 2618 ts_tablet_manager.cc:1397] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.211s user 0.020s sys 0.044s
I20260808 01:10:07.118400 2614 raft_consensus.cc:348] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:07.120040 2614 raft_consensus.cc:729] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:07.120347 2206 raft_consensus.cc:1203] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:07.121551 2614 consensus_queue.cc:260] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:07.132719 2614 ts_tablet_manager.cc:1428] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.067s user 0.012s sys 0.005s
I20260808 01:10:07.144632 1613 tablet_copy_service.cc:342] P a4862d0938c84fc4bd840e4c17acae78: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-1ccc57605ae24bf4bacee4e4741bbe95 received from {username='slave'} at 127.0.0.1:34182
I20260808 01:10:07.145109 1613 tablet_copy_service.cc:433] P a4862d0938c84fc4bd840e4c17acae78: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-1ccc57605ae24bf4bacee4e4741bbe95 on tablet 1ccc57605ae24bf4bacee4e4741bbe95 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:07.156463 2618 raft_consensus.cc:348] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:07.157752 2618 raft_consensus.cc:729] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:07.158581 2618 consensus_queue.cc:260] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:07.161607 2618 ts_tablet_manager.cc:1428] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.046s user 0.000s sys 0.010s
I20260808 01:10:07.164378 1613 tablet_copy_service.cc:342] P a4862d0938c84fc4bd840e4c17acae78: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-b3583c46ad2f4aa8875c06ef46cde18a received from {username='slave'} at 127.0.0.1:34182
I20260808 01:10:07.164830 1613 tablet_copy_service.cc:433] P a4862d0938c84fc4bd840e4c17acae78: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-b3583c46ad2f4aa8875c06ef46cde18a on tablet b3583c46ad2f4aa8875c06ef46cde18a with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:07.208344 2620 tablet_bootstrap.cc:492] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:07.209338 2620 tablet_bootstrap.cc:492] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:07.210054 2620 ts_tablet_manager.cc:1397] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.266s user 0.043s sys 0.037s
I20260808 01:10:07.212967 2620 raft_consensus.cc:348] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:07.213807 2620 raft_consensus.cc:729] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:07.214437 2620 consensus_queue.cc:260] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:07.216715 2620 ts_tablet_manager.cc:1428] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260808 01:10:07.218817 1534 tablet_copy_service.cc:342] P 6b09dbefd24b4d52b385c4dff5697dc5: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-a19c1759594d4722abadf576352f9a98 received from {username='slave'} at 127.0.0.1:36560
I20260808 01:10:07.219245 1534 tablet_copy_service.cc:433] P 6b09dbefd24b4d52b385c4dff5697dc5: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-a19c1759594d4722abadf576352f9a98 on tablet a19c1759594d4722abadf576352f9a98 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:07.351744 2206 raft_consensus.cc:1203] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:07.616483 2206 raft_consensus.cc:1203] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:07.695080 2556 raft_consensus.cc:1050] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:07.697224 2556 consensus_queue.cc:237] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:07.705662 2206 raft_consensus.cc:1261] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:07.706920 1603 raft_consensus.cc:1261] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:07.708143 1517 raft_consensus.cc:1261] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:07.708912 2628 consensus_queue.cc:1035] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:07.709914 2556 consensus_queue.cc:1035] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:07.716773 2628 consensus_queue.cc:1035] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:07.728739 2556 raft_consensus.cc:2932] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:07.733642 1604 raft_consensus.cc:2932] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:07.744030 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:07.754608 2206 raft_consensus.cc:2932] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:07.766319 2207 raft_consensus.cc:1203] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:07.778589 1342 catalog_manager.cc:5495] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:07.789896 1516 raft_consensus.cc:2932] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:07.838611 1445 consensus_queue.cc:237] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:07.850823 2206 raft_consensus.cc:1261] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:07.852589 2639 consensus_queue.cc:1035] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:07.855378 1599 raft_consensus.cc:1261] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:07.860320 2639 consensus_queue.cc:1035] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:07.866160 2587 raft_consensus.cc:2932] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:07.867776 2206 raft_consensus.cc:2932] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:07.875469 1599 raft_consensus.cc:2932] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:07.889050 1340 catalog_manager.cc:5495] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:07.903889 1511 tablet_service.cc:1514] Processing DeleteTablet for tablet 58547863eabc48eda1552ed4240f68d3 with delete_type TABLET_DATA_TOMBSTONED (TS 6b09dbefd24b4d52b385c4dff5697dc5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58250
I20260808 01:10:07.905157 1329 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 58547863eabc48eda1552ed4240f68d3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:07.916929 2646 tablet_replica.cc:330] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5: stopping tablet replica
I20260808 01:10:07.918124 2646 raft_consensus.cc:2229] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:07.919023 2646 raft_consensus.cc:2258] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:07.922577 2646 ts_tablet_manager.cc:1905] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:07.932449 2593 raft_consensus.cc:1050] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:07.934520 2593 consensus_queue.cc:237] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:07.939565 2646 ts_tablet_manager.cc:1918] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:07.940181 2646 log.cc:1196] T 58547863eabc48eda1552ed4240f68d3 P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-1-root/wals/58547863eabc48eda1552ed4240f68d3
I20260808 01:10:07.942245 1328 catalog_manager.cc:4841] TS 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): tablet 58547863eabc48eda1552ed4240f68d3 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:07.943523 2206 raft_consensus.cc:1261] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:07.945168 1445 raft_consensus.cc:1261] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:07.944950 2593 consensus_queue.cc:1035] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:07.946902 2630 consensus_queue.cc:1035] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:07.948972 1516 raft_consensus.cc:1261] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:07.950510 1445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:07.951264 1445 raft_consensus.cc:595] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Received request to transfer leadership to TS 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:07.952175 2630 consensus_queue.cc:1035] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:07.955366 1445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:07.955974 1445 raft_consensus.cc:595] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:07.968489 2630 raft_consensus.cc:2932] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:07.971565 1516 raft_consensus.cc:2932] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:07.975208 2206 raft_consensus.cc:2932] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:07.988407 1445 raft_consensus.cc:2932] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:08.002864 1603 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:08.003495 1603 raft_consensus.cc:595] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:08.005396 1341 catalog_manager.cc:5495] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:08.008347 2630 raft_consensus.cc:1050] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:08.051579 2630 consensus_queue.cc:237] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:08.055924 2587 raft_consensus.cc:979] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:08.056476 2587 raft_consensus.cc:1067] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Signalling peer 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:08.057914 2207 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:49270
I20260808 01:10:08.058457 2207 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:08.058772 2207 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:08.072194 2207 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:08.075551 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:08.076241 1517 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:08.088085 1603 consensus_queue.cc:237] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:08.089381 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:08.090001 1445 raft_consensus.cc:3032] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:08.090345 1445 raft_consensus.cc:729] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:08.091084 1445 consensus_queue.cc:260] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:08.092000 1445 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:08.088127 2207 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:08.098065 2206 raft_consensus.cc:1261] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:08.090013 1517 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:08.099025 1525 raft_consensus.cc:1226] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:08.100322 2632 consensus_queue.cc:1035] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:08.107409 2171 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:08.108564 2623 raft_consensus.cc:2781] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:08.122694 2652 raft_consensus.cc:2932] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:08.124423 2207 raft_consensus.cc:2932] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:08.130203 1525 raft_consensus.cc:1261] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:08.130463 2623 raft_consensus.cc:686] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:08.131448 2623 consensus_queue.cc:237] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:08.135929 2593 consensus_queue.cc:1035] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:08.156718 1445 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:08.182391 1517 raft_consensus.cc:2932] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:08.184123 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b3583c46ad2f4aa8875c06ef46cde18a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:08.189870 1445 raft_consensus.cc:1261] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:08.191355 2207 raft_consensus.cc:1261] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:08.194212 2593 consensus_queue.cc:1035] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:08.201153 2630 consensus_queue.cc:1035] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:08.205186 1341 catalog_manager.cc:5495] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:08.220351 1517 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:08.221294 1517 raft_consensus.cc:595] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:08.221882 1525 raft_consensus.cc:1261] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:08.225108 2652 consensus_queue.cc:1035] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:08.225580 1342 catalog_manager.cc:5495] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 1 to 2, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 2 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:08.229655 1525 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:08.230510 1525 raft_consensus.cc:595] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Received request to transfer leadership to TS 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:08.235375 1525 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:08.236259 1525 raft_consensus.cc:595] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:08.240603 1429 tablet_service.cc:1514] Processing DeleteTablet for tablet b3583c46ad2f4aa8875c06ef46cde18a with delete_type TABLET_DATA_TOMBSTONED (TS 3fb1a6380d27401a831cfe5f86a4cc9d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55036
I20260808 01:10:08.276602 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260808 01:10:08.278905 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:08.280002 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:08.288736 2665 tablet_replica.cc:330] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d: stopping tablet replica
I20260808 01:10:08.302065 2657 raft_consensus.cc:2932] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:08.308115 2607 raft_consensus.cc:1050] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:08.308960 2665 raft_consensus.cc:2229] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:08.312667 2607 consensus_queue.cc:237] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:08.324066 2665 raft_consensus.cc:2258] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:08.320716 1445 raft_consensus.cc:2932] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:08.329795 1517 raft_consensus.cc:2932] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:08.349426 2665 ts_tablet_manager.cc:1905] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:08.350270 2207 raft_consensus.cc:2932] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:08.345809 1341 catalog_manager.cc:5495] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:08.362181 1444 raft_consensus.cc:1261] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:08.368532 1596 raft_consensus.cc:1261] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:08.370337 2209 raft_consensus.cc:1261] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:08.372609 2631 consensus_queue.cc:1035] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:08.375234 2652 raft_consensus.cc:979] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:08.384016 2654 consensus_queue.cc:1035] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:08.396916 2665 ts_tablet_manager.cc:1918] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:08.397550 2665 log.cc:1196] T b3583c46ad2f4aa8875c06ef46cde18a P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root/wals/b3583c46ad2f4aa8875c06ef46cde18a
I20260808 01:10:08.398149 2607 consensus_queue.cc:1035] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:08.398773 2660 raft_consensus.cc:1067] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:08.404729 1329 catalog_manager.cc:4841] TS 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): tablet b3583c46ad2f4aa8875c06ef46cde18a (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:08.412976 1445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55040
I20260808 01:10:08.413713 1445 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:08.414170 1445 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:08.427650 1596 consensus_queue.cc:237] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:08.436780 1445 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:08.439682 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:08.440336 1516 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:08.445526 2207 raft_consensus.cc:1261] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:08.445739 1445 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:08.445927 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:08.446599 1598 raft_consensus.cc:3032] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:08.446887 1598 raft_consensus.cc:729] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:08.447175 2593 consensus_queue.cc:1035] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:08.447721 1598 consensus_queue.cc:260] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:08.449010 1598 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:08.451828 1445 raft_consensus.cc:1261] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:08.453758 2593 consensus_queue.cc:1035] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:08.461995 2593 raft_consensus.cc:2932] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:08.463948 1516 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:08.466557 1410 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:08.468057 2628 raft_consensus.cc:2781] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:08.467500 1440 raft_consensus.cc:2932] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:08.468638 2628 raft_consensus.cc:686] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:08.469540 2628 consensus_queue.cc:237] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:08.483440 2631 raft_consensus.cc:2932] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:08.493178 2207 raft_consensus.cc:2932] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:08.498497 1598 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:08.503746 1440 raft_consensus.cc:2932] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:08.516029 2608 raft_consensus.cc:979] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:08.518400 2608 raft_consensus.cc:1067] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:08.520421 1340 catalog_manager.cc:5495] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:08.533041 2672 raft_consensus.cc:979] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:08.541615 1603 raft_consensus.cc:2932] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:08.550318 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 1ccc57605ae24bf4bacee4e4741bbe95 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:08.560329 1511 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ccc57605ae24bf4bacee4e4741bbe95 with delete_type TABLET_DATA_TOMBSTONED (TS 6b09dbefd24b4d52b385c4dff5697dc5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58250
I20260808 01:10:08.591270 2628 raft_consensus.cc:1067] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Signalling peer a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:08.622166 2207 raft_consensus.cc:2932] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:10:08.631633 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 73689.9us
I20260808 01:10:08.636641 1342 catalog_manager.cc:5495] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:08.641989 1440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55042
I20260808 01:10:08.642598 1440 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:08.646523 1440 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:08.671461 1340 catalog_manager.cc:5495] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:08.675244 1599 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:45628
I20260808 01:10:08.676052 1599 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:08.676530 1599 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:08.679809 1440 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:08.681587 1440 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:08.682247 2676 tablet_replica.cc:330] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5: stopping tablet replica
I20260808 01:10:08.686842 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:08.687527 1596 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:08.690634 1599 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:08.692797 1599 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:08.693604 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:08.701205 1440 raft_consensus.cc:3032] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:08.701676 1440 raft_consensus.cc:729] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:08.704761 1596 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:08.706396 1440 consensus_queue.cc:260] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:08.709532 1440 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:08.708544 2676 raft_consensus.cc:2229] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:08.729839 1440 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:08.733661 1568 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:08.749275 2676 raft_consensus.cc:2258] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:08.751572 2630 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:08.755971 1445 raft_consensus.cc:1261] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:08.765434 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:08.766224 1516 raft_consensus.cc:3032] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:08.766547 1516 raft_consensus.cc:729] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:08.760388 2630 raft_consensus.cc:686] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:08.770524 2676 ts_tablet_manager.cc:1905] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:08.771240 2630 consensus_queue.cc:237] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:08.767275 1516 consensus_queue.cc:260] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:08.783192 1516 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:10:08.788698 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 105824us
W20260808 01:10:08.789541 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 99891.6us
I20260808 01:10:08.799504 2681 raft_consensus.cc:979] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:08.810678 2543 raft_consensus.cc:1067] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Signalling peer 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:08.820587 2207 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:49268
I20260808 01:10:08.821430 2207 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:08.821851 2207 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:08.810750 1516 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:08.837387 2680 consensus_queue.cc:1035] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260808 01:10:08.841629 2207 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:08.843860 2207 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:10:08.844762 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 86798.4us
I20260808 01:10:08.853731 1516 raft_consensus.cc:1261] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:08.854341 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:08.855564 1444 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:08.856415 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:08.857378 1516 raft_consensus.cc:3032] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:08.857714 1516 raft_consensus.cc:729] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:08.860908 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:08.861593 1517 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:08.861735 1410 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:08.867228 1516 consensus_queue.cc:260] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:08.870489 1516 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:08.875634 1525 consensus_queue.cc:237] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:08.885469 2680 consensus_queue.cc:1035] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:08.900206 2685 raft_consensus.cc:2781] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:08.922382 2207 raft_consensus.cc:1261] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:08.928960 1444 raft_consensus.cc:2454] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:10:08.918362 2685 raft_consensus.cc:686] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:08.952665 2682 consensus_queue.cc:1035] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260808 01:10:08.952893 2685 consensus_queue.cc:237] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:08.952117 1517 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:08.956431 1516 raft_consensus.cc:2454] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:10:08.961930 2172 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:08.964350 2680 raft_consensus.cc:2781] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:08.964880 2680 raft_consensus.cc:686] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:08.965837 2680 consensus_queue.cc:237] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:08.999301 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 121887us
W20260808 01:10:09.017268 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 87123.3us
I20260808 01:10:09.046239 2687 raft_consensus.cc:979] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:09.053691 2607 raft_consensus.cc:2932] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:09.064478 2209 raft_consensus.cc:2932] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:09.068668 2687 raft_consensus.cc:1067] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:09.063027 2676 ts_tablet_manager.cc:1918] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
W20260808 01:10:09.079279 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 61161.4us
W20260808 01:10:09.081096 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 53215.9us
I20260808 01:10:09.089744 2676 log.cc:1196] T 1ccc57605ae24bf4bacee4e4741bbe95 P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-1-root/wals/1ccc57605ae24bf4bacee4e4741bbe95
I20260808 01:10:09.093515 1328 catalog_manager.cc:4841] TS 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): tablet 1ccc57605ae24bf4bacee4e4741bbe95 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:09.174568 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a19c1759594d4722abadf576352f9a98 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:10:09.181100 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 117497us
I20260808 01:10:09.186117 1342 catalog_manager.cc:5495] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:09.185799 1339 catalog_manager.cc:5495] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 3 to 4, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 4 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 01:10:09.200985 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 277198us
I20260808 01:10:09.208564 1525 raft_consensus.cc:1261] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:09.229048 2679 consensus_queue.cc:1035] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260808 01:10:09.249471 1603 raft_consensus.cc:1261] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:09.251932 1339 catalog_manager.cc:5495] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 1 to 2, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 2 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:10:09.285732 1448 raft_consensus.cc:1261] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:09.290072 1443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55042
I20260808 01:10:09.290627 1443 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:09.290985 1443 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:09.302479 1448 raft_consensus.cc:1261] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:09.305830 2693 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:09.309958 1586 tablet_service.cc:1514] Processing DeleteTablet for tablet a19c1759594d4722abadf576352f9a98 with delete_type TABLET_DATA_TOMBSTONED (TS a4862d0938c84fc4bd840e4c17acae78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45612
I20260808 01:10:09.323376 1525 raft_consensus.cc:1261] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:09.325104 2697 consensus_queue.cc:1035] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:09.325516 1443 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:09.326560 1448 raft_consensus.cc:1226] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:09.339790 1443 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:09.353366 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:09.354831 1525 raft_consensus.cc:3032] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:09.356146 1525 raft_consensus.cc:729] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:09.365944 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:09.366652 1599 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:09.363735 1525 consensus_queue.cc:260] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:09.369391 1525 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:09.370759 2687 consensus_queue.cc:1035] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:09.368193 1339 catalog_manager.cc:5495] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:09.409426 1599 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:09.411054 1410 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:09.413836 1443 raft_consensus.cc:2932] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:09.458024 1525 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:09.428589 2639 raft_consensus.cc:2781] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:09.418596 2660 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20260808 01:10:09.540877 2639 raft_consensus.cc:686] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
W20260808 01:10:09.556017 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 194547us
I20260808 01:10:09.543061 2639 consensus_queue.cc:237] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:09.569977 2706 tablet_replica.cc:330] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78: stopping tablet replica
I20260808 01:10:09.591606 2706 raft_consensus.cc:2229] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:09.593369 2706 raft_consensus.cc:2258] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:09.598799 2706 ts_tablet_manager.cc:1905] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:09.658411 1444 raft_consensus.cc:1261] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:10:09.714800 2623 consensus_queue.cc:1035] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260808 01:10:09.798548 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260808 01:10:09.900842 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 56058.2us
I20260808 01:10:09.902441 1516 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260808 01:10:09.935122 1443 consensus_meta.cc:287] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.513s user 0.007s sys 0.012s
I20260808 01:10:09.935534 2719 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:09.942055 1519 raft_consensus.cc:1261] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260808 01:10:09.946120 1490 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.515s user 0.042s sys 0.087s
W20260808 01:10:09.947540 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 516597us
I20260808 01:10:09.965858 2623 consensus_queue.cc:1035] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260808 01:10:09.972607 1517 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.049s sys 0.077s
W20260808 01:10:09.973187 1517 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s user 0.050s sys 0.077s
I20260808 01:10:09.981211 1599 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260808 01:10:10.011430 2672 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.020s sys 0.033s
W20260808 01:10:10.051921 2672 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.573s user 0.022s sys 0.033s
W20260808 01:10:10.085367 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 182001us
I20260808 01:10:10.124440 2717 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
W20260808 01:10:10.135046 2718 log.cc:924] Time spent T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.062s user 0.001s sys 0.002s
I20260808 01:10:10.231539 1519 raft_consensus.cc:1261] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:10.278841 2706 ts_tablet_manager.cc:1918] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:10.279604 2706 log.cc:1196] T a19c1759594d4722abadf576352f9a98 P a4862d0938c84fc4bd840e4c17acae78: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-2-root/wals/a19c1759594d4722abadf576352f9a98
I20260808 01:10:10.300129 2697 consensus_queue.cc:1035] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260808 01:10:10.273219 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 51203.6us
I20260808 01:10:10.322346 1328 catalog_manager.cc:4841] TS a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): tablet a19c1759594d4722abadf576352f9a98 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
W20260808 01:10:10.336944 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 69793.5us
I20260808 01:10:10.420773 1340 catalog_manager.cc:5495] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
W20260808 01:10:10.422679 2739 log.cc:924] Time spent T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20260808 01:10:10.483942 1519 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:10.484980 1519 raft_consensus.cc:595] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Received request to transfer leadership to TS 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:10.489377 1440 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:10.489936 1440 raft_consensus.cc:595] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Received request to transfer leadership to TS 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:10.501317 1519 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:10.503716 1519 raft_consensus.cc:595] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Received request to transfer leadership to TS 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:10.515786 1440 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:10.516513 1440 raft_consensus.cc:595] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Received request to transfer leadership to TS 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:10.520036 1519 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:10.520668 1519 raft_consensus.cc:595] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:10.526216 1604 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:10.526906 1604 raft_consensus.cc:595] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:10.528452 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260808 01:10:10.530170 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:10.545776 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:10.568089 2692 raft_consensus.cc:979] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:10.568732 2692 raft_consensus.cc:1067] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Signalling peer 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:10.571154 2205 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:49270
I20260808 01:10:10.571815 2205 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:10.572257 2205 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:10.583216 2205 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:10.588382 2687 raft_consensus.cc:979] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:10.588934 2687 raft_consensus.cc:1067] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Signalling peer 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:10.591454 1604 consensus_queue.cc:237] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:10.604946 2209 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:49268
I20260808 01:10:10.605635 2209 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:10.605990 2209 raft_consensus.cc:3037] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:10.619181 2205 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:10.620297 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:10.621420 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:10.621750 1597 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:10.622001 1440 raft_consensus.cc:3032] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:10.622249 1440 raft_consensus.cc:729] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:10.625996 1440 consensus_queue.cc:260] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:10.626873 1440 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:10.639046 1597 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:10.640553 2171 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:10.640897 2209 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:10.641670 2701 raft_consensus.cc:2781] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:10.642143 2701 raft_consensus.cc:686] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:10.643155 2209 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:10.649991 1440 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:10.653126 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:10.653715 1519 raft_consensus.cc:3032] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:10.654050 1519 raft_consensus.cc:729] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
W20260808 01:10:10.654266 1565 consensus_peers.cc:487] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73a11329d12c4ba4abd2b4e0e29816c7. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:10.655045 1519 consensus_queue.cc:260] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:10.656368 1519 raft_consensus.cc:3037] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:10.661976 1522 raft_consensus.cc:1261] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:10.664533 2630 raft_consensus.cc:979] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:10.643080 2701 consensus_queue.cc:237] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:10.666893 2693 consensus_queue.cc:1035] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:10.667944 2630 raft_consensus.cc:1067] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:10.669696 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:10.670008 1445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55040
I20260808 01:10:10.670351 1443 raft_consensus.cc:3037] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:10.670504 1445 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:10.670836 1445 raft_consensus.cc:3037] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:10.685214 1440 raft_consensus.cc:1261] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:10.686501 1445 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:10.687000 2630 consensus_queue.cc:1035] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:10.699322 1445 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:10.691349 1519 raft_consensus.cc:2454] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:10.689047 1443 raft_consensus.cc:2454] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:10.700618 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:10.706054 1597 raft_consensus.cc:3032] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:10.706501 1597 raft_consensus.cc:729] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:10.707506 1597 consensus_queue.cc:260] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:10.709360 1597 raft_consensus.cc:3037] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:10.702009 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:10.710387 1525 raft_consensus.cc:3037] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:10.715327 2172 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:10.722256 1342 catalog_manager.cc:5495] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 1 to 2, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 2 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:10.724112 2684 raft_consensus.cc:2932] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:10.720505 2701 raft_consensus.cc:2781] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:10.727281 2701 raft_consensus.cc:686] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:10.728194 1525 raft_consensus.cc:2454] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:10.728363 2701 consensus_queue.cc:237] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:10.730198 1447 raft_consensus.cc:2932] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:10.731057 1522 raft_consensus.cc:2932] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:10.738960 1410 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:10.744184 2679 raft_consensus.cc:2781] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:10.744711 2679 raft_consensus.cc:686] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:10.745576 2679 consensus_queue.cc:237] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:10.759320 1339 catalog_manager.cc:5495] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:10.764392 1597 raft_consensus.cc:2454] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:10.778177 1447 consensus_queue.cc:237] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:10.811295 1339 catalog_manager.cc:5495] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 2 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 01:10:10.824059 1408 consensus_peers.cc:487] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 846d38a10c5846789f0b926b61905c0b. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:10.824633 1519 raft_consensus.cc:1261] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:10.825809 1596 raft_consensus.cc:1261] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:10.826537 2679 consensus_queue.cc:1035] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:10.830817 2697 consensus_queue.cc:1035] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:10.830384 1340 catalog_manager.cc:5495] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:10.844038 2697 raft_consensus.cc:2932] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:10.850412 1525 raft_consensus.cc:2932] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:10.861591 1597 raft_consensus.cc:2932] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:10.864161 1519 consensus_queue.cc:237] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:10.877099 1339 catalog_manager.cc:5495] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
W20260808 01:10:10.887092 1487 consensus_peers.cc:487] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b1ad14dc361495f8e43a49a35526563. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:10.889935 1443 raft_consensus.cc:1261] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:10.890986 1597 raft_consensus.cc:1261] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:10.891853 2712 consensus_queue.cc:1035] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:10.894487 2712 consensus_queue.cc:1035] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:10.905674 2607 raft_consensus.cc:2932] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:10.908085 1596 raft_consensus.cc:2932] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:10.910671 1443 raft_consensus.cc:2932] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:10.931034 1443 consensus_queue.cc:237] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:10.927675 1339 catalog_manager.cc:5495] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 2 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:10.940165 1596 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:10.942783 2679 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260808 01:10:10.944046 1408 consensus_peers.cc:487] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc9cccb55e8444778a9b2833e3d89a5a. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:10.945775 1519 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:10.949039 2685 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:10.962564 2685 raft_consensus.cc:2932] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:10.972275 1525 raft_consensus.cc:2932] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:10.970808 1596 raft_consensus.cc:2932] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:10.980512 2697 raft_consensus.cc:979] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:10.981122 2697 raft_consensus.cc:1067] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Signalling peer 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:10.984385 2209 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:49270
I20260808 01:10:10.984978 2209 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:10.985337 2209 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:10.995142 2741 raft_consensus.cc:979] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:10.995653 2741 raft_consensus.cc:1067] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Signalling peer 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:10.993546 1342 catalog_manager.cc:5495] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:11.001278 2209 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:11.002699 2206 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:49268
I20260808 01:10:11.003268 2209 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:11.003286 2206 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:11.004556 2206 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:11.009492 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:11.010064 1597 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:11.011122 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:11.011709 1443 raft_consensus.cc:3032] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:11.012033 1443 raft_consensus.cc:729] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:11.012864 1443 consensus_queue.cc:260] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:11.014195 1443 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:11.016757 1597 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:11.017781 2171 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:11.018883 2701 raft_consensus.cc:2781] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:11.019388 2701 raft_consensus.cc:686] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:11.020452 2701 consensus_queue.cc:237] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:11.025429 1443 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:11.030534 2206 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:11.032395 2712 raft_consensus.cc:979] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:11.032902 2712 raft_consensus.cc:1067] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:11.033227 2206 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:11.034546 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:11.035156 1604 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:11.030953 1339 catalog_manager.cc:5495] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 1 to 2, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 2 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:11.036298 1443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55042
I20260808 01:10:11.036839 1443 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:11.037171 1443 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:11.045255 1443 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:11.047761 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:11.048436 1519 raft_consensus.cc:3032] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:11.048556 1604 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:11.048844 1519 raft_consensus.cc:729] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:11.049609 2171 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:11.049701 1519 consensus_queue.cc:260] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:11.050906 1519 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:11.052284 1443 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:11.054569 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:11.055123 1604 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:11.058744 1519 raft_consensus.cc:2454] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:11.059892 2701 raft_consensus.cc:2781] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:11.060349 1410 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:11.061203 2697 raft_consensus.cc:2781] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:11.061692 2697 raft_consensus.cc:686] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:11.061362 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:11.060714 2701 raft_consensus.cc:686] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:11.062601 2697 consensus_queue.cc:237] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:11.063333 1525 raft_consensus.cc:3032] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:11.066931 1445 raft_consensus.cc:1261] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:11.068105 1604 raft_consensus.cc:2454] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:11.069037 2737 consensus_queue.cc:1035] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:11.067348 1525 raft_consensus.cc:729] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:11.072620 1525 consensus_queue.cc:260] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:11.076411 1525 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:11.065178 2701 consensus_queue.cc:237] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:11.103214 1339 catalog_manager.cc:5495] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 2 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:11.112042 1525 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:11.120179 1597 raft_consensus.cc:1261] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:11.125694 2737 consensus_queue.cc:1035] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:11.148612 1342 catalog_manager.cc:5495] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:11.170326 2766 ts_tablet_manager.cc:927] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:11.172935 2766 tablet_copy_client.cc:316] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.67:35675
I20260808 01:10:11.174985 1613 tablet_copy_service.cc:140] P a4862d0938c84fc4bd840e4c17acae78: Received BeginTabletCopySession request for tablet 73a11329d12c4ba4abd2b4e0e29816c7 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:34182)
I20260808 01:10:11.175638 1613 tablet_copy_service.cc:161] P a4862d0938c84fc4bd840e4c17acae78: Beginning new tablet copy session on tablet 73a11329d12c4ba4abd2b4e0e29816c7 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:34182: session id = 95b661b03f1d460b9657e61d4b0247cd-73a11329d12c4ba4abd2b4e0e29816c7
I20260808 01:10:11.187508 1613 tablet_copy_source_session.cc:205] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:11.190922 2766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a11329d12c4ba4abd2b4e0e29816c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:11.191964 1525 raft_consensus.cc:1261] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:11.197798 2737 consensus_queue.cc:1035] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:11.216555 2766 tablet_copy_client.cc:791] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:11.217278 2766 tablet_copy_client.cc:655] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:11.223338 2766 tablet_copy_client.cc:531] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:11.227406 1447 raft_consensus.cc:1261] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:11.231899 2701 consensus_queue.cc:1035] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:11.246675 2766 tablet_bootstrap.cc:492] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:11.289976 2766 tablet_bootstrap.cc:492] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:11.291057 2766 tablet_bootstrap.cc:492] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:11.291625 2766 ts_tablet_manager.cc:1397] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.045s user 0.022s sys 0.006s
I20260808 01:10:11.294939 2766 raft_consensus.cc:348] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:11.296746 2766 raft_consensus.cc:729] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:11.297386 2766 consensus_queue.cc:260] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:11.300002 2766 ts_tablet_manager.cc:1428] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.008s user 0.007s sys 0.004s
I20260808 01:10:11.301913 1613 tablet_copy_service.cc:342] P a4862d0938c84fc4bd840e4c17acae78: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-73a11329d12c4ba4abd2b4e0e29816c7 received from {username='slave'} at 127.0.0.1:34182
I20260808 01:10:11.302376 1613 tablet_copy_service.cc:433] P a4862d0938c84fc4bd840e4c17acae78: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-73a11329d12c4ba4abd2b4e0e29816c7 on tablet 73a11329d12c4ba4abd2b4e0e29816c7 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:11.317925 2766 ts_tablet_manager.cc:927] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
I20260808 01:10:11.320267 2766 tablet_copy_client.cc:316] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.65:38291
I20260808 01:10:11.322218 1458 tablet_copy_service.cc:140] P 3fb1a6380d27401a831cfe5f86a4cc9d: Received BeginTabletCopySession request for tablet 846d38a10c5846789f0b926b61905c0b from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:60392)
I20260808 01:10:11.322748 1458 tablet_copy_service.cc:161] P 3fb1a6380d27401a831cfe5f86a4cc9d: Beginning new tablet copy session on tablet 846d38a10c5846789f0b926b61905c0b from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:60392: session id = 95b661b03f1d460b9657e61d4b0247cd-846d38a10c5846789f0b926b61905c0b
I20260808 01:10:11.329008 1604 raft_consensus.cc:1261] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:11.330490 2685 consensus_queue.cc:1035] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:11.334229 1458 tablet_copy_source_session.cc:205] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:11.345690 2766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 846d38a10c5846789f0b926b61905c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:11.355392 1519 raft_consensus.cc:1261] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:11.359023 2697 consensus_queue.cc:1035] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:11.372092 2766 tablet_copy_client.cc:791] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:11.372766 2766 tablet_copy_client.cc:655] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:11.382058 2766 tablet_copy_client.cc:531] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:11.392803 2766 tablet_bootstrap.cc:492] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:11.412493 1596 raft_consensus.cc:1261] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:11.413897 2769 consensus_queue.cc:1035] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:11.420570 2766 tablet_bootstrap.cc:492] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:11.429069 1445 raft_consensus.cc:1261] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:11.431056 2769 consensus_queue.cc:1035] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:11.435286 2779 ts_tablet_manager.cc:927] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
I20260808 01:10:11.436254 2766 tablet_bootstrap.cc:492] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:11.437800 2766 ts_tablet_manager.cc:1397] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.018s
I20260808 01:10:11.441547 2766 raft_consensus.cc:348] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:11.448329 2766 raft_consensus.cc:729] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:11.449568 2779 tablet_copy_client.cc:316] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.65:38291
I20260808 01:10:11.449577 2766 consensus_queue.cc:260] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:11.451952 1458 tablet_copy_service.cc:140] P 3fb1a6380d27401a831cfe5f86a4cc9d: Received BeginTabletCopySession request for tablet bc9cccb55e8444778a9b2833e3d89a5a from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:60392)
I20260808 01:10:11.452610 1458 tablet_copy_service.cc:161] P 3fb1a6380d27401a831cfe5f86a4cc9d: Beginning new tablet copy session on tablet bc9cccb55e8444778a9b2833e3d89a5a from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:60392: session id = 95b661b03f1d460b9657e61d4b0247cd-bc9cccb55e8444778a9b2833e3d89a5a
I20260808 01:10:11.454958 2766 ts_tablet_manager.cc:1428] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20260808 01:10:11.467233 1457 tablet_copy_service.cc:342] P 3fb1a6380d27401a831cfe5f86a4cc9d: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-846d38a10c5846789f0b926b61905c0b received from {username='slave'} at 127.0.0.1:60392
I20260808 01:10:11.467919 1457 tablet_copy_service.cc:433] P 3fb1a6380d27401a831cfe5f86a4cc9d: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-846d38a10c5846789f0b926b61905c0b on tablet 846d38a10c5846789f0b926b61905c0b with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:11.473032 1522 raft_consensus.cc:1261] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:11.478514 2766 ts_tablet_manager.cc:927] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:11.482612 2717 consensus_queue.cc:1035] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:11.496781 1458 tablet_copy_source_session.cc:205] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:11.515300 2766 tablet_copy_client.cc:316] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.66:37093
I20260808 01:10:11.517230 1534 tablet_copy_service.cc:140] P 6b09dbefd24b4d52b385c4dff5697dc5: Received BeginTabletCopySession request for tablet 6b1ad14dc361495f8e43a49a35526563 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:36560)
I20260808 01:10:11.517843 1534 tablet_copy_service.cc:161] P 6b09dbefd24b4d52b385c4dff5697dc5: Beginning new tablet copy session on tablet 6b1ad14dc361495f8e43a49a35526563 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:36560: session id = 95b661b03f1d460b9657e61d4b0247cd-6b1ad14dc361495f8e43a49a35526563
I20260808 01:10:11.526683 2779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9cccb55e8444778a9b2833e3d89a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:11.538805 1534 tablet_copy_source_session.cc:205] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:11.543993 2766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b1ad14dc361495f8e43a49a35526563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:11.545533 1597 raft_consensus.cc:1261] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:11.547318 1604 raft_consensus.cc:1261] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:11.553539 2697 consensus_queue.cc:1035] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:11.556417 2778 consensus_queue.cc:1035] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:11.588302 2766 tablet_copy_client.cc:791] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:11.589012 2766 tablet_copy_client.cc:655] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:11.605834 1522 raft_consensus.cc:1261] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:11.609949 2778 consensus_queue.cc:1035] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:11.614061 2766 tablet_copy_client.cc:531] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:11.619030 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:11.624346 2206 raft_consensus.cc:1203] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:11.636463 2779 tablet_copy_client.cc:791] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:11.637221 2779 tablet_copy_client.cc:655] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:11.657498 2779 tablet_copy_client.cc:531] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:11.680472 2766 tablet_bootstrap.cc:492] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:11.719859 2779 tablet_bootstrap.cc:492] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:11.749794 2766 tablet_bootstrap.cc:492] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:11.750761 2766 tablet_bootstrap.cc:492] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:11.751426 2766 ts_tablet_manager.cc:1397] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.071s user 0.050s sys 0.013s
I20260808 01:10:11.754894 2209 raft_consensus.cc:1203] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:11.753986 2766 raft_consensus.cc:348] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:11.755530 2766 raft_consensus.cc:729] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:11.756157 2766 consensus_queue.cc:260] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:11.758463 2766 ts_tablet_manager.cc:1428] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 01:10:11.764107 1534 tablet_copy_service.cc:342] P 6b09dbefd24b4d52b385c4dff5697dc5: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-6b1ad14dc361495f8e43a49a35526563 received from {username='slave'} at 127.0.0.1:36560
I20260808 01:10:11.764653 1534 tablet_copy_service.cc:433] P 6b09dbefd24b4d52b385c4dff5697dc5: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-6b1ad14dc361495f8e43a49a35526563 on tablet 6b1ad14dc361495f8e43a49a35526563 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:11.798789 2779 tablet_bootstrap.cc:492] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:11.799057 1445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:11.799798 1445 raft_consensus.cc:595] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:11.799944 2779 tablet_bootstrap.cc:492] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:11.801038 2779 ts_tablet_manager.cc:1397] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.081s user 0.025s sys 0.023s
I20260808 01:10:11.803244 1443 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:11.803989 1443 raft_consensus.cc:595] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Received request to transfer leadership to TS 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:11.804489 2779 raft_consensus.cc:348] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:11.805560 2779 raft_consensus.cc:729] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:11.806347 2779 consensus_queue.cc:260] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:11.807919 1445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:11.808460 1445 raft_consensus.cc:595] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:11.810842 2779 ts_tablet_manager.cc:1428] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260808 01:10:11.811062 1525 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:11.812094 1525 raft_consensus.cc:595] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:11.813220 1458 tablet_copy_service.cc:342] P 3fb1a6380d27401a831cfe5f86a4cc9d: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-bc9cccb55e8444778a9b2833e3d89a5a received from {username='slave'} at 127.0.0.1:60392
I20260808 01:10:11.813632 1458 tablet_copy_service.cc:433] P 3fb1a6380d27401a831cfe5f86a4cc9d: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-bc9cccb55e8444778a9b2833e3d89a5a on tablet bc9cccb55e8444778a9b2833e3d89a5a with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:11.817775 1525 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:11.818432 1525 raft_consensus.cc:595] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:11.819595 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260808 01:10:11.821996 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:11.822660 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:11.916452 2697 raft_consensus.cc:979] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:11.917040 2685 raft_consensus.cc:1067] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Signalling peer a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:11.919239 1604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:45628
I20260808 01:10:11.919718 1604 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:11.920131 1604 raft_consensus.cc:3037] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:11.928731 2717 raft_consensus.cc:979] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:11.929296 2717 raft_consensus.cc:1067] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Signalling peer a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:11.939714 2209 raft_consensus.cc:1203] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260808 01:10:11.931298 1597 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:45628
I20260808 01:10:11.941709 1597 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:11.942113 1597 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:11.947855 2209 raft_consensus.cc:1203] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20260808 01:10:11.965328 1604 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:11.968513 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:11.969022 1443 raft_consensus.cc:3032] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:11.969283 1443 raft_consensus.cc:729] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:11.969950 1443 consensus_queue.cc:260] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:11.971007 1443 raft_consensus.cc:3037] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:11.972976 2697 raft_consensus.cc:979] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:11.973464 2697 raft_consensus.cc:1067] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Signalling peer 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:11.976256 2209 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:49270
I20260808 01:10:11.976846 2209 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:11.977419 2209 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:11.978003 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:11.978641 1522 raft_consensus.cc:3037] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:11.986673 2209 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:11.989333 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:11.989889 1445 raft_consensus.cc:3032] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:11.991694 1443 raft_consensus.cc:2454] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:11.993108 1568 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:11.993755 1445 raft_consensus.cc:729] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:11.994287 2684 raft_consensus.cc:2781] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:11.994628 1445 consensus_queue.cc:260] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:11.995338 1522 raft_consensus.cc:2454] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:11.995934 1445 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:11.997284 2684 raft_consensus.cc:686] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:11.998971 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:11.999504 2209 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:11.999603 1596 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:11.998685 2684 consensus_queue.cc:237] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:12.004923 1445 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:12.006129 2172 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:12.007041 2769 raft_consensus.cc:2781] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:12.007520 2769 raft_consensus.cc:686] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:12.008433 2769 consensus_queue.cc:237] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:12.011489 1596 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:12.017699 1597 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:12.019872 1597 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:12.022012 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:12.022681 1525 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:12.023990 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:12.024746 1445 raft_consensus.cc:3032] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:12.025116 1445 raft_consensus.cc:729] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:12.026371 1445 consensus_queue.cc:260] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:12.027420 1445 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:12.031746 1339 catalog_manager.cc:5495] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 1 to 2, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 2 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:12.035882 1525 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:12.036157 1445 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:12.037497 1567 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:12.038502 1340 catalog_manager.cc:5495] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 1 to 2, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 2 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:10:12.039629 2684 raft_consensus.cc:2781] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:12.040367 2684 raft_consensus.cc:686] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:12.041249 2684 consensus_queue.cc:237] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:12.043275 1604 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:12.074473 2632 raft_consensus.cc:1050] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:12.077343 2632 consensus_queue.cc:237] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:12.081936 1340 catalog_manager.cc:5495] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 2 to 3, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 3 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:10:12.108263 2792 raft_consensus.cc:979] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:12.108836 2792 raft_consensus.cc:1067] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:12.110780 1443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55042
I20260808 01:10:12.111398 1443 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:12.111853 1443 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:12.119648 1443 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:12.121800 1443 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:12.124691 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:12.125748 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:12.126443 1522 raft_consensus.cc:3032] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:12.126746 1522 raft_consensus.cc:729] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:12.127091 1525 raft_consensus.cc:1261] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:12.129477 2209 raft_consensus.cc:1261] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:12.127563 1522 consensus_queue.cc:260] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:12.131620 1522 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:12.136240 1443 raft_consensus.cc:1261] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:12.139215 2684 consensus_queue.cc:1035] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:12.141721 2684 consensus_queue.cc:1035] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:12.144804 2630 consensus_queue.cc:1035] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:12.125828 1597 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:12.153735 1522 raft_consensus.cc:2454] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:12.166446 1597 raft_consensus.cc:2454] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:12.169135 1410 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:12.170562 2717 raft_consensus.cc:2781] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:12.171105 2717 raft_consensus.cc:686] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:12.172919 2717 consensus_queue.cc:237] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:12.173960 2632 raft_consensus.cc:2932] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.190698 2712 raft_consensus.cc:979] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:12.191293 2712 raft_consensus.cc:1067] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Signalling peer a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:12.194916 1598 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:45638
I20260808 01:10:12.189220 1443 raft_consensus.cc:2932] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.195487 1598 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:12.195928 1598 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:12.202734 2685 raft_consensus.cc:1050] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:12.204368 1598 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:12.206521 2685 consensus_queue.cc:237] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:12.209110 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:12.209844 1522 raft_consensus.cc:3032] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:12.210192 1522 raft_consensus.cc:729] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:12.212204 1522 consensus_queue.cc:260] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:12.211064 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:12.213395 2209 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:12.210728 1598 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:12.218350 1522 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:12.223527 2206 raft_consensus.cc:2932] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.226387 1522 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:12.232105 2208 raft_consensus.cc:1261] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:12.232417 1567 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:12.233850 2630 raft_consensus.cc:2781] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:12.234221 1598 raft_consensus.cc:1261] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:12.234305 2717 consensus_queue.cc:1035] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:12.236022 1522 raft_consensus.cc:2932] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.234361 2630 raft_consensus.cc:686] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:12.238423 2630 consensus_queue.cc:237] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:12.241927 2697 consensus_queue.cc:1035] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:12.223858 1340 catalog_manager.cc:5495] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:12.248940 2209 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:12.261170 1519 raft_consensus.cc:1261] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:12.260665 1339 catalog_manager.cc:5495] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.272080 2717 consensus_queue.cc:1035] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:12.309401 1597 consensus_queue.cc:237] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:12.319109 2685 raft_consensus.cc:2932] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.325399 1516 raft_consensus.cc:2932] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.335425 1522 raft_consensus.cc:1261] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:12.335889 2206 raft_consensus.cc:1261] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:12.338025 2684 consensus_queue.cc:1035] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:12.341063 2632 consensus_queue.cc:1035] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:12.340018 1339 catalog_manager.cc:5495] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 2 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:12.345772 2630 raft_consensus.cc:2932] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.336910 1597 raft_consensus.cc:2932] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.350540 1522 raft_consensus.cc:2932] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.359128 1342 catalog_manager.cc:5495] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.367545 2206 raft_consensus.cc:2932] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.388425 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 73a11329d12c4ba4abd2b4e0e29816c7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:12.380681 2208 raft_consensus.cc:2932] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.395125 1341 catalog_manager.cc:5495] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.398272 1598 raft_consensus.cc:1261] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:12.400235 2769 consensus_queue.cc:1035] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:12.407696 1429 tablet_service.cc:1514] Processing DeleteTablet for tablet 73a11329d12c4ba4abd2b4e0e29816c7 with delete_type TABLET_DATA_TOMBSTONED (TS 3fb1a6380d27401a831cfe5f86a4cc9d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55036
I20260808 01:10:12.417240 1445 consensus_queue.cc:237] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:12.422194 2812 tablet_replica.cc:330] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d: stopping tablet replica
I20260808 01:10:12.423684 2812 raft_consensus.cc:2229] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:12.424649 2812 raft_consensus.cc:2258] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:12.430083 1447 raft_consensus.cc:1261] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:12.432389 2630 consensus_queue.cc:1035] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:12.434783 2209 raft_consensus.cc:1261] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:12.440409 2685 consensus_queue.cc:1035] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:12.443094 1522 raft_consensus.cc:1261] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:12.445978 2679 consensus_queue.cc:1035] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:12.453197 2812 ts_tablet_manager.cc:1905] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:12.458284 1447 raft_consensus.cc:1261] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:12.460220 2769 consensus_queue.cc:1035] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:12.466722 2679 raft_consensus.cc:2932] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.468293 1516 raft_consensus.cc:2932] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.477154 2206 raft_consensus.cc:2932] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.482909 1329 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 846d38a10c5846789f0b926b61905c0b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:12.491557 1522 raft_consensus.cc:1261] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:12.492990 2812 ts_tablet_manager.cc:1918] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:12.493633 2812 log.cc:1196] T 73a11329d12c4ba4abd2b4e0e29816c7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root/wals/73a11329d12c4ba4abd2b4e0e29816c7
I20260808 01:10:12.493693 2630 consensus_queue.cc:1035] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:12.494228 1439 raft_consensus.cc:1261] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:12.496609 2630 consensus_queue.cc:1035] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:12.495931 1329 catalog_manager.cc:4841] TS 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): tablet 73a11329d12c4ba4abd2b4e0e29816c7 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:12.525333 2697 raft_consensus.cc:1050] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:12.527498 2697 consensus_queue.cc:237] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:12.537248 1339 catalog_manager.cc:5495] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.542306 1516 raft_consensus.cc:1261] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:12.543918 2630 consensus_queue.cc:1035] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:12.574951 1604 raft_consensus.cc:1261] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:12.578158 2209 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:12.582192 1604 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:12.582886 2679 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:12.582238 1586 tablet_service.cc:1514] Processing DeleteTablet for tablet 846d38a10c5846789f0b926b61905c0b with delete_type TABLET_DATA_TOMBSTONED (TS a4862d0938c84fc4bd840e4c17acae78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45612
I20260808 01:10:12.585263 2697 consensus_queue.cc:1035] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:12.586292 2679 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:12.594995 2687 raft_consensus.cc:1050] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:12.595644 1520 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:12.602398 2679 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:12.604432 2824 tablet_replica.cc:330] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78: stopping tablet replica
I20260808 01:10:12.597889 2687 consensus_queue.cc:237] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:12.645882 2824 raft_consensus.cc:2229] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:12.654426 2824 raft_consensus.cc:2258] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:12.669392 2209 raft_consensus.cc:1261] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:12.671041 2792 consensus_queue.cc:1035] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:12.741272 2824 ts_tablet_manager.cc:1905] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:12.749927 2697 raft_consensus.cc:2932] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.752786 1522 raft_consensus.cc:2932] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.763512 1520 raft_consensus.cc:1261] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:12.775259 1604 raft_consensus.cc:2932] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.786479 1516 raft_consensus.cc:1261] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:12.791342 2806 consensus_queue.cc:1035] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:12.805179 1604 raft_consensus.cc:1261] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:12.812335 2741 consensus_queue.cc:1035] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:12.835119 1445 raft_consensus.cc:1261] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:12.837625 2684 consensus_queue.cc:1035] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:12.838835 2824 ts_tablet_manager.cc:1918] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:12.840245 2209 raft_consensus.cc:2932] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.849177 2824 log.cc:1196] T 846d38a10c5846789f0b926b61905c0b P a4862d0938c84fc4bd840e4c17acae78: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-2-root/wals/846d38a10c5846789f0b926b61905c0b
I20260808 01:10:12.853446 1328 catalog_manager.cc:4841] TS a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): tablet 846d38a10c5846789f0b926b61905c0b (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:12.855295 2687 consensus_queue.cc:1035] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:12.864164 1340 catalog_manager.cc:5495] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:12.868335 2206 raft_consensus.cc:1261] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:12.858054 2687 raft_consensus.cc:2932] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.874121 2800 consensus_queue.cc:1035] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:12.883314 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:12.885928 1447 consensus_queue.cc:237] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:12.892343 2208 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:12.894430 2685 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:12.868361 1604 raft_consensus.cc:2932] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.946046 2697 raft_consensus.cc:2932] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:10:12.951452 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 84549.7us
I20260808 01:10:12.958294 2205 raft_consensus.cc:2932] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.960547 2207 raft_consensus.cc:2932] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.963918 1598 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:12.976387 1329 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet bc9cccb55e8444778a9b2833e3d89a5a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:12.984864 1340 catalog_manager.cc:5495] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:12.987244 2697 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:13.010294 1342 catalog_manager.cc:5495] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index 4 to 5, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:13.033944 1604 raft_consensus.cc:2932] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:13.036363 1447 raft_consensus.cc:2932] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:13.046248 1511 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9cccb55e8444778a9b2833e3d89a5a with delete_type TABLET_DATA_TOMBSTONED (TS 6b09dbefd24b4d52b385c4dff5697dc5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58250
I20260808 01:10:13.061769 2840 tablet_replica.cc:330] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5: stopping tablet replica
I20260808 01:10:13.066728 2840 raft_consensus.cc:2229] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:13.067552 2840 raft_consensus.cc:2258] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:13.074136 2840 ts_tablet_manager.cc:1905] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:13.082821 1522 consensus_queue.cc:237] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:13.108669 2206 raft_consensus.cc:1261] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:13.117118 2741 consensus_queue.cc:1035] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:13.130620 1604 raft_consensus.cc:1261] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:13.133311 2830 raft_consensus.cc:2932] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:13.146919 2206 raft_consensus.cc:2932] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:13.141940 2792 consensus_queue.cc:1035] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:13.162961 1604 raft_consensus.cc:2932] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:13.163894 2840 ts_tablet_manager.cc:1918] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260808 01:10:13.166833 2840 log.cc:1196] T bc9cccb55e8444778a9b2833e3d89a5a P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-1-root/wals/bc9cccb55e8444778a9b2833e3d89a5a
I20260808 01:10:13.179646 1339 catalog_manager.cc:5495] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New cstate: current_term: 2 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:13.183081 1328 catalog_manager.cc:4841] TS 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): tablet bc9cccb55e8444778a9b2833e3d89a5a (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:13.193960 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 6b1ad14dc361495f8e43a49a35526563 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:13.202862 1429 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b1ad14dc361495f8e43a49a35526563 with delete_type TABLET_DATA_TOMBSTONED (TS 3fb1a6380d27401a831cfe5f86a4cc9d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55036
I20260808 01:10:13.208736 2841 tablet_replica.cc:330] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d: stopping tablet replica
I20260808 01:10:13.210522 2841 raft_consensus.cc:2229] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:13.211560 2841 raft_consensus.cc:2258] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:13.221417 1522 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:13.222081 1522 raft_consensus.cc:595] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Received request to transfer leadership to TS 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:13.223455 2841 ts_tablet_manager.cc:1905] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:13.226054 1522 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:13.226778 1522 raft_consensus.cc:595] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:13.233906 1445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:13.234604 1445 raft_consensus.cc:595] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Received request to transfer leadership to TS 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:13.241789 1522 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a83a963773094d6db6495b7009fd8690"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:13.242461 1522 raft_consensus.cc:595] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:13.246068 1604 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:13.246603 1604 raft_consensus.cc:595] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:13.247896 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260808 01:10:13.249330 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:13.249976 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:13.262004 2841 ts_tablet_manager.cc:1918] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260808 01:10:13.262763 2841 log.cc:1196] T 6b1ad14dc361495f8e43a49a35526563 P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root/wals/6b1ad14dc361495f8e43a49a35526563
I20260808 01:10:13.266251 1329 catalog_manager.cc:4841] TS 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): tablet 6b1ad14dc361495f8e43a49a35526563 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:13.424650 2838 raft_consensus.cc:979] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:13.425186 2800 raft_consensus.cc:1067] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:13.426718 1520 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58280
I20260808 01:10:13.427189 1520 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:13.427531 1520 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:13.434623 1520 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:13.436465 1520 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:13.437489 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:13.438112 1598 raft_consensus.cc:3032] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:13.438439 1598 raft_consensus.cc:729] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:13.439287 1598 consensus_queue.cc:260] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:13.440590 1598 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:13.441668 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:13.442425 2206 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:13.446909 1598 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:13.448791 1489 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:13.448964 2206 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:13.449831 2830 raft_consensus.cc:2781] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:13.450531 2830 raft_consensus.cc:686] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:13.452275 2830 consensus_queue.cc:237] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:13.463671 2687 raft_consensus.cc:979] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:13.464740 2687 raft_consensus.cc:1067] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Signalling peer 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:13.468664 2206 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:49268
I20260808 01:10:13.469281 2206 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:13.469664 2206 raft_consensus.cc:3037] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:13.481240 2206 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:13.483106 2206 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:13.484700 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:13.485572 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:13.486333 1598 raft_consensus.cc:3037] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:13.491397 1338 catalog_manager.cc:5495] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 1 to 2, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 2 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:13.494100 2792 raft_consensus.cc:979] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:13.494671 2712 raft_consensus.cc:1067] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:13.497462 1445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a83a963773094d6db6495b7009fd8690"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55042
I20260808 01:10:13.498098 1445 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:13.498458 1445 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:13.503612 1598 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:13.485585 1520 raft_consensus.cc:3032] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:13.504801 1520 raft_consensus.cc:729] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:13.504897 2171 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:13.505803 2769 raft_consensus.cc:2781] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:13.506323 2769 raft_consensus.cc:686] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:13.507246 2769 consensus_queue.cc:237] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:13.504252 1445 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:13.513851 1445 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:13.516079 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:13.516285 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:13.513883 1520 consensus_queue.cc:260] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:13.516636 1522 raft_consensus.cc:3032] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:13.516845 1598 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:13.517040 1522 raft_consensus.cc:729] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:13.517413 1520 raft_consensus.cc:3037] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:13.517890 1522 consensus_queue.cc:260] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:13.518942 1522 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:13.525763 1520 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:13.525944 1522 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:13.527845 1598 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:13.530174 1410 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:13.530953 2685 raft_consensus.cc:2781] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:13.531373 2685 raft_consensus.cc:686] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:13.532241 2685 consensus_queue.cc:237] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:13.538969 1339 catalog_manager.cc:5495] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 2 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:13.554490 1339 catalog_manager.cc:5495] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 3 to 4, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 4 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:13.593395 2687 raft_consensus.cc:979] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:13.593928 2830 raft_consensus.cc:1067] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:13.595443 1445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55042
I20260808 01:10:13.595964 1445 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:13.596195 1445 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:13.602430 1445 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:13.604709 1445 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:13.605420 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:13.605996 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:13.606101 1522 raft_consensus.cc:3032] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:13.606596 1522 raft_consensus.cc:729] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:13.606596 1598 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:13.607468 1522 consensus_queue.cc:260] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:13.608451 1522 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:13.612269 1598 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:13.614147 1522 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:13.614288 1410 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:13.615206 2685 raft_consensus.cc:2781] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:13.615680 2685 raft_consensus.cc:686] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:13.616474 2685 consensus_queue.cc:237] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:13.626331 1339 catalog_manager.cc:5495] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:13.661527 2831 raft_consensus.cc:979] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:13.662091 2685 raft_consensus.cc:1067] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Signalling peer 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:13.663763 2209 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:49270
I20260808 01:10:13.664366 2209 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:13.664747 2209 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:13.671439 2209 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:13.673337 2209 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:13.674118 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:13.674508 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:13.674760 1447 raft_consensus.cc:3032] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:13.675060 1447 raft_consensus.cc:729] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:13.675060 1522 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:13.675904 1447 consensus_queue.cc:260] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:13.677156 1447 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:13.681946 1522 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:13.683193 2171 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:13.684218 2769 raft_consensus.cc:2781] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:13.684547 1447 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:13.684904 2769 raft_consensus.cc:686] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:13.686044 2769 consensus_queue.cc:237] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:13.704780 1339 catalog_manager.cc:5495] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 1 to 2, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 2 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:13.892583 1598 raft_consensus.cc:1261] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:13.893800 2769 consensus_queue.cc:1035] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:13.904592 1522 raft_consensus.cc:1261] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:13.911119 2769 consensus_queue.cc:1035] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:13.945199 1598 raft_consensus.cc:1261] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:13.946468 2687 consensus_queue.cc:1035] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:13.956264 2209 raft_consensus.cc:1261] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:13.958220 2830 consensus_queue.cc:1035] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:13.986007 1597 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:10:13.987349 2685 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:13.998461 1522 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:10:13.999588 2685 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:14.114217 1522 raft_consensus.cc:1261] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:14.115477 2685 consensus_queue.cc:1035] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:14.124855 1598 raft_consensus.cc:1261] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:14.127643 2831 consensus_queue.cc:1035] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:14.183591 1447 raft_consensus.cc:1261] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:14.185357 2769 consensus_queue.cc:1035] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:14.197362 1522 raft_consensus.cc:1261] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:14.199402 2851 consensus_queue.cc:1035] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:14.283381 1522 consensus_queue.cc:237] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:14.289223 1445 raft_consensus.cc:1261] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 01:10:14.289532 1487 consensus_peers.cc:487] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82bb58b6f7c14c22814cfac6d2b2bf6d. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:14.289783 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:14.290159 1598 raft_consensus.cc:1261] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:14.291450 2792 consensus_queue.cc:1035] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:14.301189 2792 consensus_queue.cc:1035] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 01:10:14.340006 2864 log.cc:924] Time spent T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20260808 01:10:14.341972 2792 raft_consensus.cc:2932] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:14.488605 1598 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:14.489291 1598 raft_consensus.cc:595] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:14.527217 1520 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:14.528039 1520 raft_consensus.cc:595] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:14.530972 1525 consensus_queue.cc:237] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:14.543444 1443 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a83a963773094d6db6495b7009fd8690"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:14.544155 1443 raft_consensus.cc:595] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:14.555483 2687 raft_consensus.cc:979] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:14.556934 1443 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:14.557721 1443 raft_consensus.cc:595] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:14.559309 2873 raft_consensus.cc:1067] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:14.559993 1445 raft_consensus.cc:2932] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:14.565913 1447 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55042
I20260808 01:10:14.567129 1443 raft_consensus.cc:1261] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:14.571789 2876 consensus_queue.cc:1035] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260808 01:10:14.599080 1447 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:14.599639 1447 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:14.613198 2876 raft_consensus.cc:2932] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:14.614848 1447 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:14.618098 1443 raft_consensus.cc:2932] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:14.647583 1443 consensus_queue.cc:237] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:14.656490 1525 raft_consensus.cc:1261] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:14.659123 2871 consensus_queue.cc:1035] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:14.658881 1338 catalog_manager.cc:5495] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:14.677704 2883 ts_tablet_manager.cc:927] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:10:14.681416 1408 consensus_peers.cc:487] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e0a7f6aa072473098cb2c51da278a83. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:14.702472 2883 tablet_copy_client.cc:316] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.66:37093
I20260808 01:10:14.701208 2831 raft_consensus.cc:2932] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:14.703347 1525 raft_consensus.cc:2932] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:14.710261 1534 tablet_copy_service.cc:140] P 6b09dbefd24b4d52b385c4dff5697dc5: Received BeginTabletCopySession request for tablet 82bb58b6f7c14c22814cfac6d2b2bf6d from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:36560)
I20260808 01:10:14.711261 1534 tablet_copy_service.cc:161] P 6b09dbefd24b4d52b385c4dff5697dc5: Beginning new tablet copy session on tablet 82bb58b6f7c14c22814cfac6d2b2bf6d from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:36560: session id = 95b661b03f1d460b9657e61d4b0247cd-82bb58b6f7c14c22814cfac6d2b2bf6d
I20260808 01:10:14.732353 1534 tablet_copy_source_session.cc:205] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:14.738245 2883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82bb58b6f7c14c22814cfac6d2b2bf6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:14.740211 1447 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:14.739960 1443 consensus_queue.cc:237] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:14.749046 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:14.749867 1525 raft_consensus.cc:3032] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:14.750295 1525 raft_consensus.cc:729] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:14.752810 1525 consensus_queue.cc:260] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:10:14.753791 1408 consensus_peers.cc:487] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e354fdfbf2264912b39cd20be2cb19f8. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:14.754287 1525 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:14.753820 1522 raft_consensus.cc:1261] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:14.761449 2871 consensus_queue.cc:1035] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:14.764540 1525 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:14.765796 1410 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
W20260808 01:10:14.766155 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 268107us
I20260808 01:10:14.766958 2871 raft_consensus.cc:2781] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:14.767854 2871 raft_consensus.cc:686] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:14.772274 2871 consensus_queue.cc:237] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:10:14.778954 1487 consensus_peers.cc:487] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a2611aecd26485e8912ab55890ec0fa. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:14.823360 1598 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:14.824128 1598 raft_consensus.cc:595] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:14.837023 1604 raft_consensus.cc:2932] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:14.865504 1598 raft_consensus.cc:1261] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:14.863178 1597 raft_consensus.cc:1261] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:14.880662 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:14.881423 1596 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:14.880949 2871 raft_consensus.cc:2932] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:14.885552 1525 raft_consensus.cc:2932] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:14.889374 1594 raft_consensus.cc:1261] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:14.897349 2838 raft_consensus.cc:979] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:14.899789 2866 raft_consensus.cc:1067] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:14.923951 2685 raft_consensus.cc:979] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:14.924518 2685 raft_consensus.cc:1067] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Signalling peer a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:14.931795 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260808 01:10:14.933858 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:14.934561 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:14.938661 1596 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:14.942965 1447 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55040
I20260808 01:10:14.943572 1447 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:14.944118 1447 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:14.961863 1599 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:45628
I20260808 01:10:14.962567 1599 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:14.962908 1599 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:14.964488 1447 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:14.968308 2697 consensus_queue.cc:1035] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:14.970721 1447 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:14.967424 2876 consensus_queue.cc:1035] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260808 01:10:14.980724 2685 consensus_queue.cc:1035] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:14.992166 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:14.993312 1525 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:10:15.017403 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 53222us
I20260808 01:10:15.030243 1525 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
W20260808 01:10:15.035063 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 65976.3us
I20260808 01:10:15.048650 1339 catalog_manager.cc:5495] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:15.050731 1339 catalog_manager.cc:5495] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:15.052657 2877 raft_consensus.cc:979] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:15.053293 2877 raft_consensus.cc:1067] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:15.059480 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:15.060141 1596 raft_consensus.cc:3032] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:15.060324 1525 raft_consensus.cc:1226] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:15.060443 1596 raft_consensus.cc:729] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:15.061497 1596 consensus_queue.cc:260] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:15.062979 1596 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:15.067664 1339 catalog_manager.cc:5495] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:15.080467 1447 raft_consensus.cc:1226] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:15.086944 1410 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:15.087878 1599 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:15.095883 1598 raft_consensus.cc:1226] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:15.096546 1599 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:15.097990 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:15.098992 2209 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:15.069821 2883 tablet_copy_client.cc:791] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:15.101403 1604 raft_consensus.cc:2932] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
W20260808 01:10:15.104022 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 60460.2us
I20260808 01:10:15.104527 1596 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:15.106416 1525 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a83a963773094d6db6495b7009fd8690"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58266
I20260808 01:10:15.106950 1525 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:15.107300 1525 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:15.114895 2883 tablet_copy_client.cc:655] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:15.117409 2868 raft_consensus.cc:2781] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:15.118000 2868 raft_consensus.cc:686] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:15.118906 2868 consensus_queue.cc:237] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:15.126797 1525 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:15.134724 1525 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:15.152329 2898 ts_tablet_manager.cc:927] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:15.159422 2209 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:15.164860 2206 raft_consensus.cc:1226] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:15.165798 1565 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:15.175060 2883 tablet_copy_client.cc:531] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:15.190001 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:15.190883 1443 raft_consensus.cc:3032] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:10:15.191242 1443 raft_consensus.cc:729] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:15.193608 1443 consensus_queue.cc:260] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:15.195115 1443 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:15.225973 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:15.226939 1599 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:15.244921 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:15.245595 1447 raft_consensus.cc:3032] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:15.245870 1447 raft_consensus.cc:729] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:15.246749 1443 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:10:15.246637 1447 consensus_queue.cc:260] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:15.247710 1447 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:15.236196 2892 raft_consensus.cc:2781] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:15.250806 1594 raft_consensus.cc:2932] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:15.287367 1447 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:15.250380 2898 tablet_copy_client.cc:316] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.66:37093
I20260808 01:10:15.294677 1599 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
W20260808 01:10:15.305357 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 63443.5us
W20260808 01:10:15.308707 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 66501.2us
I20260808 01:10:15.310061 2895 ts_tablet_manager.cc:927] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
I20260808 01:10:15.312906 1598 raft_consensus.cc:2932] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:15.321108 1490 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:15.328454 2892 raft_consensus.cc:686] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:15.329665 2892 consensus_queue.cc:237] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:15.336150 2866 raft_consensus.cc:979] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:15.348855 1534 tablet_copy_service.cc:140] P 6b09dbefd24b4d52b385c4dff5697dc5: Received BeginTabletCopySession request for tablet 7a2611aecd26485e8912ab55890ec0fa from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:36560)
I20260808 01:10:15.349474 1534 tablet_copy_service.cc:161] P 6b09dbefd24b4d52b385c4dff5697dc5: Beginning new tablet copy session on tablet 7a2611aecd26485e8912ab55890ec0fa from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:36560: session id = 95b661b03f1d460b9657e61d4b0247cd-7a2611aecd26485e8912ab55890ec0fa
I20260808 01:10:15.373967 2687 raft_consensus.cc:2781] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:10:15.374563 2687 raft_consensus.cc:686] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:15.375319 2687 consensus_queue.cc:237] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:10:15.379077 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 58093.6us
I20260808 01:10:15.393671 2895 tablet_copy_client.cc:316] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.65:38291
I20260808 01:10:15.399752 2910 raft_consensus.cc:1067] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:15.391125 2883 tablet_bootstrap.cc:492] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
W20260808 01:10:15.440060 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 83819.7us
I20260808 01:10:15.442464 1458 tablet_copy_service.cc:140] P 3fb1a6380d27401a831cfe5f86a4cc9d: Received BeginTabletCopySession request for tablet 4e0a7f6aa072473098cb2c51da278a83 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:60392)
I20260808 01:10:15.443097 1458 tablet_copy_service.cc:161] P 3fb1a6380d27401a831cfe5f86a4cc9d: Beginning new tablet copy session on tablet 4e0a7f6aa072473098cb2c51da278a83 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:60392: session id = 95b661b03f1d460b9657e61d4b0247cd-4e0a7f6aa072473098cb2c51da278a83
W20260808 01:10:15.459867 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 159466us
I20260808 01:10:15.466145 1520 raft_consensus.cc:1261] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260808 01:10:15.463163 2743 log.cc:924] Time spent T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23: Append to log took a long time: real 0.092s user 0.001s sys 0.001s
W20260808 01:10:15.524020 1547 debug-util.cc:398] Leaking SignalData structure 0x7b0800171280 after lost signal to thread 32596
I20260808 01:10:15.527981 1520 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58280
W20260808 01:10:15.623042 1547 debug-util.cc:398] Leaking SignalData structure 0x7b0800171960 after lost signal to thread 1376
I20260808 01:10:15.634441 2914 ts_tablet_manager.cc:927] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
W20260808 01:10:15.758962 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 293493us
W20260808 01:10:15.802030 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 273762us
I20260808 01:10:15.793099 1534 tablet_copy_source_session.cc:205] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:15.528626 1520 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:15.813509 1520 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:15.803927 1458 tablet_copy_source_session.cc:205] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:15.825534 1598 raft_consensus.cc:1261] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:15.835521 1520 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:15.840996 1520 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:10:15.843541 1547 debug-util.cc:398] Leaking SignalData structure 0x7b0800351680 after lost signal to thread 1473
W20260808 01:10:15.845172 1547 debug-util.cc:398] Leaking SignalData structure 0x7b0800340a20 after lost signal to thread 1551
I20260808 01:10:15.846170 2883 tablet_bootstrap.cc:700] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd: Time spent opening tablet: real 0.109s user 0.002s sys 0.002s
I20260808 01:10:15.813097 2868 consensus_queue.cc:1035] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
W20260808 01:10:15.847968 1547 debug-util.cc:398] Leaking SignalData structure 0x7b080022e9a0 after lost signal to thread 1629
W20260808 01:10:15.910537 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 46493.3us
W20260808 01:10:15.913599 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 88408us
I20260808 01:10:15.922047 1525 raft_consensus.cc:1261] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260808 01:10:15.960951 1547 debug-util.cc:398] Leaking SignalData structure 0x7b0800342600 after lost signal to thread 2233
W20260808 01:10:16.010239 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 194261us
I20260808 01:10:16.014066 1525 raft_consensus.cc:1226] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:16.049922 2697 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:16.058292 2915 consensus_queue.cc:1035] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260808 01:10:16.081210 2910 consensus_queue.cc:1046] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:16.084940 2910 raft_consensus.cc:3032] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:16.093964 2898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2611aecd26485e8912ab55890ec0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:16.110917 1440 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:16.111234 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:16.112596 1438 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:16.118865 1443 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:10:16.124478 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:16.145797 1594 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:10:16.103423 2910 raft_consensus.cc:729] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:16.156601 2910 consensus_queue.cc:260] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:16.158835 2910 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:16.171940 2928 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260808 01:10:16.199729 2911 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
W20260808 01:10:16.288405 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 83665.5us
I20260808 01:10:16.285024 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:16.312899 2207 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:16.386435 2923 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
W20260808 01:10:16.413978 1408 consensus_peers.cc:487] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:16.472347 2872 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.170s
I20260808 01:10:16.449812 1438 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:16.516438 2895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0a7f6aa072473098cb2c51da278a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:16.519861 1599 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:16.553237 1339 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55752 (request call id 210) took 1703 ms. Trace:
I20260808 01:10:16.557266 1342 catalog_manager.cc:5495] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
W20260808 01:10:16.561561 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 71292.7us
I20260808 01:10:16.588492 2898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:49268 (request call id 407) took 1555 ms. Trace:
I20260808 01:10:16.589274 2898 rpcz_store.cc:276] 0808 01:10:15.033047 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:15.033188 (+ 141us) service_pool.cc:227] Handling call
0808 01:10:15.152862 (+119674us) ts_tablet_manager.cc:928] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
0808 01:10:16.091462 (+938600us) tablet_copy_client.cc:361] Tablet copy session begun
0808 01:10:16.329508 (+238046us) tablet_metadata.cc:641] Metadata flushed
0808 01:10:16.352765 (+ 23257us) tablet_copy_client.cc:490] Wrote new tablet metadata
0808 01:10:16.587088 (+234323us) tablet_copy_client.cc:411] Wrote new consensus metadata
0808 01:10:16.587574 (+ 486us) tablet_replica.cc:205] Creating consensus instance
0808 01:10:16.588186 (+ 612us) ts_tablet_manager.cc:983] Replica registered.
0808 01:10:16.588316 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":21999,"tablet-copy.queue_time_us":113388,"thread_start_us":205644,"threads_started":2}
I20260808 01:10:16.599674 1447 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55036
I20260808 01:10:16.600546 1447 raft_consensus.cc:595] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Received request to transfer leadership
I20260808 01:10:16.593837 2898 tablet_copy_client.cc:791] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
W20260808 01:10:16.612926 2914 thread.cc:636] tablet-download-pool [worker] (thread pool) Time spent creating pthread: real 0.729s user 0.054s sys 0.153s
W20260808 01:10:16.649781 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 345756us
W20260808 01:10:16.655350 1598 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.024s sys 0.042s
W20260808 01:10:16.655670 1598 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.517s user 0.024s sys 0.042s
I20260808 01:10:16.646907 2898 tablet_copy_client.cc:655] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:16.553824 1339 rpcz_store.cc:276] 0808 01:10:14.849744 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:14.849920 (+ 176us) service_pool.cc:227] Handling call
0808 01:10:14.868106 (+ 18186us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b5400002dd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4af0 00007fe34ca47f2a 00007fe352999d6c 00007fe352a93c3e 00007fe350da5fb4 00007fe350da5f42 00007fe350da5e71 00007fe350da5dec 00007fe350da50a2 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:16.553081 (+1684975us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 01:10:15.156832 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:10:15.263846 (+107014us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:10:15.267616 (+ 3770us) write_op.cc:435] Acquired schema lock
0808 01:10:15.267634 (+ 18us) tablet.cc:660] Decoding operations
0808 01:10:15.323490 (+ 55856us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:10:15.323556 (+ 66us) write_op.cc:641] Partition lock acquired for write op
0808 01:10:15.323586 (+ 30us) tablet.cc:683] Acquiring locks for 3 operations
0808 01:10:15.329657 (+ 6071us) tablet.cc:699] Row locks acquired
0808 01:10:15.329676 (+ 19us) write_op.cc:260] PREPARE: finished
0808 01:10:15.329856 (+ 180us) write_op.cc:270] Start()
0808 01:10:15.329964 (+ 108us) write_op.cc:276] Timestamp: P: 1786151415329812 usec, L: 0
0808 01:10:15.329987 (+ 23us) op_driver.cc:331] REPLICATION: starting
0808 01:10:15.370335 (+ 40348us) log.cc:842] Serialized 1491 byte log entry
0808 01:10:16.010391 (+640056us) op_driver.cc:444] REPLICATION: finished
0808 01:10:16.392681 (+382290us) write_op.cc:301] APPLY: starting
0808 01:10:16.396858 (+ 4177us) tablet.cc:1365] starting BulkCheckPresence
0808 01:10:16.424857 (+ 27999us) tablet.cc:1368] finished BulkCheckPresence
0808 01:10:16.425172 (+ 315us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 01:10:16.431765 (+ 6593us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 01:10:16.432045 (+ 280us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0808 01:10:16.432145 (+ 100us) write_op.cc:312] APPLY: finished
0808 01:10:16.484974 (+ 52829us) log.cc:842] Serialized 35 byte log entry
0808 01:10:16.498955 (+ 13981us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:10:16.499103 (+ 148us) write_op.cc:454] Released schema lock
0808 01:10:16.500717 (+ 1614us) write_op.cc:341] FINISH: Updating metrics
0808 01:10:16.501288 (+ 571us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:10:16.501336 (+ 48us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":4,"spinlock_wait_cycles":30357248,"child_traces":[["op",{"apply.queue_time_us":326984,"apply.run_cpu_time_us":5415,"apply.run_wall_time_us":124481,"num_ops":3,"prepare.queue_time_us":4306,"prepare.run_cpu_time_us":4827,"prepare.run_wall_time_us":237094,"replication_time_us":651862,"thread_start_us":166761,"threads_started":1}]]}
I20260808 01:10:16.652513 1489 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:10:16.653725 2914 thread.cc:602] tablet-download-pool [worker] (thread pool) Time spent starting thread: real 0.784s user 0.055s sys 0.153s
I20260808 01:10:16.647148 2922 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:16.689356 1596 raft_consensus.cc:1261] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:16.697363 2905 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:16.711509 2877 raft_consensus.cc:979] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:16.712607 2877 raft_consensus.cc:1067] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:16.705175 2922 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:16.730273 2905 raft_consensus.cc:686] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:16.729347 2914 tablet_copy_client.cc:316] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.65:38291
I20260808 01:10:16.783986 2685 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 01:10:16.799257 2940 log.cc:924] Time spent T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20260808 01:10:16.816074 1525 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58266
I20260808 01:10:16.841127 2922 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:16.841656 1442 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:16.843194 1442 raft_consensus.cc:595] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:10:16.819458 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 335466us
I20260808 01:10:16.882479 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:16.848623 2898 tablet_copy_client.cc:531] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:16.924176 1594 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bc8c41cadc374229baf248366012f202"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:16.924976 1594 raft_consensus.cc:595] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:16.963201 1339 catalog_manager.cc:5495] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 2 to 3, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 3 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:16.827764 2905 consensus_queue.cc:237] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:16.912637 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:10:17.041651 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 82506.2us
I20260808 01:10:16.961230 1442 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51830
W20260808 01:10:17.052026 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 86935.1us
I20260808 01:10:17.027391 1458 tablet_copy_service.cc:140] P 3fb1a6380d27401a831cfe5f86a4cc9d: Received BeginTabletCopySession request for tablet e354fdfbf2264912b39cd20be2cb19f8 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:60392)
I20260808 01:10:17.077639 1458 tablet_copy_service.cc:161] P 3fb1a6380d27401a831cfe5f86a4cc9d: Beginning new tablet copy session on tablet e354fdfbf2264912b39cd20be2cb19f8 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:60392: session id = 95b661b03f1d460b9657e61d4b0247cd-e354fdfbf2264912b39cd20be2cb19f8
W20260808 01:10:17.096630 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 71027.3us
I20260808 01:10:17.075034 1442 raft_consensus.cc:595] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:17.127652 1568 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:17.158704 2866 raft_consensus.cc:2726] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:17.172380 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260808 01:10:17.174646 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:17.214605 2883 tablet_bootstrap.cc:492] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:17.215437 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:17.238770 1458 tablet_copy_source_session.cc:205] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:17.277838 2883 tablet_bootstrap.cc:492] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:17.279439 2883 ts_tablet_manager.cc:1397] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 1.904s user 0.037s sys 0.103s
W20260808 01:10:17.294994 2207 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.002s sys 0.072s
W20260808 01:10:17.295538 2207 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.541s user 0.002s sys 0.073s
I20260808 01:10:17.385565 2895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:49270 (request call id 188) took 2276 ms. Trace:
W20260808 01:10:17.390630 2948 log.cc:924] Time spent T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.249s user 0.002s sys 0.005s
I20260808 01:10:17.418027 2932 raft_consensus.cc:979] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:17.418834 2932 raft_consensus.cc:1067] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:17.432688 2931 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
W20260808 01:10:17.399286 2877 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.004s sys 0.027s
W20260808 01:10:17.453922 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 403078us
I20260808 01:10:17.433660 2931 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:17.479249 1525 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:17.479900 1525 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:10:17.452965 2877 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s user 0.005s sys 0.027s
I20260808 01:10:17.386366 2895 rpcz_store.cc:276] 0808 01:10:15.109371 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:15.110226 (+ 855us) service_pool.cc:227] Handling call
0808 01:10:15.310570 (+200344us) ts_tablet_manager.cc:928] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
0808 01:10:16.340129 (+1029559us) tablet_copy_client.cc:361] Tablet copy session begun
0808 01:10:16.811476 (+471347us) tablet_metadata.cc:641] Metadata flushed
0808 01:10:16.811745 (+ 269us) tablet_copy_client.cc:490] Wrote new tablet metadata
0808 01:10:17.294514 (+482769us) tablet_copy_client.cc:411] Wrote new consensus metadata
0808 01:10:17.320501 (+ 25987us) tablet_replica.cc:205] Creating consensus instance
0808 01:10:17.349123 (+ 28622us) ts_tablet_manager.cc:983] Replica registered.
0808 01:10:17.355647 (+ 6524us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":17033,"tablet-copy.queue_time_us":86127,"thread_start_us":103111,"threads_started":2}
I20260808 01:10:17.481662 1442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bc8c41cadc374229baf248366012f202"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55040
I20260808 01:10:17.505152 2895 tablet_copy_client.cc:791] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:17.518462 1525 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:17.538967 1447 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55036
I20260808 01:10:17.539705 1447 raft_consensus.cc:595] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Received request to transfer leadership
I20260808 01:10:17.549139 2895 tablet_copy_client.cc:655] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:17.549585 1442 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:17.576117 1442 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:10:17.555927 1329 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 82bb58b6f7c14c22814cfac6d2b2bf6d already in progress [suppressed 2 similar messages]
W20260808 01:10:17.548815 2743 log.cc:924] Time spent T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23: Append to log took a long time: real 0.062s user 0.001s sys 0.001s
I20260808 01:10:17.591789 2931 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:17.594310 2883 raft_consensus.cc:348] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:17.629799 2883 raft_consensus.cc:729] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
W20260808 01:10:17.632019 2870 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.003s sys 0.025s
W20260808 01:10:17.632601 2870 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.725s user 0.004s sys 0.025s
I20260808 01:10:17.633517 1525 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:17.630988 2883 consensus_queue.cc:260] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:17.659086 2914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e354fdfbf2264912b39cd20be2cb19f8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:10:17.659942 1410 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.004s sys 0.022s
W20260808 01:10:17.660398 1410 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.627s user 0.004s sys 0.022s
W20260808 01:10:17.654336 2838 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.021s sys 0.045s
W20260808 01:10:17.662640 2685 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.638s user 0.023s sys 0.068s
W20260808 01:10:17.690567 2685 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.666s user 0.023s sys 0.069s
W20260808 01:10:17.690429 2838 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s user 0.021s sys 0.046s
I20260808 01:10:17.785889 2898 tablet_bootstrap.cc:492] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:17.787585 2685 raft_consensus.cc:979] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:17.788273 2931 raft_consensus.cc:1067] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:17.792102 2931 raft_consensus.cc:979] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:17.792914 2871 raft_consensus.cc:1067] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
W20260808 01:10:17.838474 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 308111us
I20260808 01:10:17.854205 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:17.855305 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:17.875516 1520 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58266
I20260808 01:10:17.877897 1520 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:17.878350 1520 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:17.893857 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:17.894538 1603 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:17.889483 2883 ts_tablet_manager.cc:1428] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.594s user 0.008s sys 0.008s
I20260808 01:10:17.913561 1603 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:17.929373 1489 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:17.930656 1520 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:10:17.914731 2883 ts_tablet_manager.cc:1460] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd: Tablet startup took 618ms
I20260808 01:10:17.908718 2970 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
W20260808 01:10:17.935731 1329 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 379846us
W20260808 01:10:17.933002 2968 log.cc:924] Time spent T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260808 01:10:17.947834 1520 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:17.938077 1442 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:17.954784 2207 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Advancing to term 2
I20260808 01:10:17.951462 1518 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58266
I20260808 01:10:17.956579 1518 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:17.957100 1518 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:10:17.935277 2883 ts_tablet_manager.cc:1462] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd: Trace:
0808 01:10:14.667105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:14.669573 (+ 2468us) service_pool.cc:227] Handling call
0808 01:10:14.682344 (+ 12771us) ts_tablet_manager.cc:928] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
0808 01:10:14.736155 (+ 53811us) tablet_copy_client.cc:361] Tablet copy session begun
0808 01:10:14.772008 (+ 35853us) tablet_metadata.cc:641] Metadata flushed
0808 01:10:14.815101 (+ 43093us) tablet_copy_client.cc:490] Wrote new tablet metadata
0808 01:10:15.016044 (+200943us) tablet_copy_client.cc:411] Wrote new consensus metadata
0808 01:10:15.064017 (+ 47973us) tablet_replica.cc:205] Creating consensus instance
0808 01:10:15.064588 (+ 571us) ts_tablet_manager.cc:983] Replica registered.
0808 01:10:15.064659 (+ 71us) inbound_call.cc:173] Queueing success response
0808 01:10:15.368975 (+304316us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0808 01:10:17.296697 (+1927722us) ts_tablet_manager.cc:1429] Starting tablet replica
0808 01:10:17.482201 (+185504us) tablet_replica.cc:259] Starting instrumentation
0808 01:10:17.576167 (+ 93966us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":2862,"mutex_wait_us":18679,"raft.queue_time_us":2519,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":205,"tablet-copy.queue_time_us":7373,"tablet-download-pool.queue_time_us":1589,"tablet-download-pool.run_cpu_time_us":1194,"tablet-download-pool.run_wall_time_us":57821,"thread_start_us":23202,"threads_started":2}
I20260808 01:10:17.962147 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:17.962713 1603 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:17.970753 1603 raft_consensus.cc:2454] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:17.975436 1489 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:17.979957 1443 raft_consensus.cc:1226] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:17.951718 2971 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:17.980717 1534 tablet_copy_service.cc:342] P 6b09dbefd24b4d52b385c4dff5697dc5: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-82bb58b6f7c14c22814cfac6d2b2bf6d received from {username='slave'} at 127.0.0.1:36560
I20260808 01:10:17.989053 2929 raft_consensus.cc:2781] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:17.989290 1447 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55036
I20260808 01:10:18.006282 1447 raft_consensus.cc:595] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Received request to transfer leadership
I20260808 01:10:17.942045 2970 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:18.005616 1534 tablet_copy_service.cc:433] P 6b09dbefd24b4d52b385c4dff5697dc5: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-82bb58b6f7c14c22814cfac6d2b2bf6d on tablet 82bb58b6f7c14c22814cfac6d2b2bf6d with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:18.010264 2970 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:17.994798 1442 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:17.968042 2961 raft_consensus.cc:2781] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:18.011795 2895 tablet_copy_client.cc:531] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:18.011948 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:18.002465 2971 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:10:18.012898 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 121181us
I20260808 01:10:18.003943 1518 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:18.012079 2961 raft_consensus.cc:686] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:18.016072 2929 raft_consensus.cc:686] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:18.043462 1447 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55036
I20260808 01:10:18.045245 1447 raft_consensus.cc:595] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Received request to transfer leadership
I20260808 01:10:18.055070 2929 consensus_queue.cc:237] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:18.060784 2914 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:49270 (request call id 190) took 2715 ms. Trace:
I20260808 01:10:18.068463 1342 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55756 (request call id 241) took 2932 ms. Trace:
I20260808 01:10:18.069051 1342 rpcz_store.cc:276] 0808 01:10:15.132958 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:15.133175 (+ 217us) service_pool.cc:227] Handling call
0808 01:10:18.065304 (+2932129us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 01:10:16.779699 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:10:16.854430 (+ 74731us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:10:16.885593 (+ 31163us) write_op.cc:435] Acquired schema lock
0808 01:10:16.888291 (+ 2698us) tablet.cc:660] Decoding operations
0808 01:10:16.959879 (+ 71588us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:10:17.026328 (+ 66449us) write_op.cc:641] Partition lock acquired for write op
0808 01:10:17.028865 (+ 2537us) tablet.cc:683] Acquiring locks for 1 operations
0808 01:10:17.051097 (+ 22232us) tablet.cc:699] Row locks acquired
0808 01:10:17.058915 (+ 7818us) write_op.cc:260] PREPARE: finished
0808 01:10:17.108824 (+ 49909us) write_op.cc:270] Start()
0808 01:10:17.119126 (+ 10302us) write_op.cc:276] Timestamp: P: 1786151417098822 usec, L: 0
0808 01:10:17.122901 (+ 3775us) op_driver.cc:331] REPLICATION: starting
0808 01:10:17.182735 (+ 59834us) log.cc:842] Serialized 599 byte log entry
0808 01:10:17.569749 (+387014us) op_driver.cc:444] REPLICATION: finished
0808 01:10:17.951365 (+381616us) write_op.cc:301] APPLY: starting
0808 01:10:17.951455 (+ 90us) tablet.cc:1365] starting BulkCheckPresence
0808 01:10:17.951556 (+ 101us) tablet.cc:1368] finished BulkCheckPresence
0808 01:10:17.956447 (+ 4891us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 01:10:17.956697 (+ 250us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 01:10:17.956851 (+ 154us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 01:10:17.956904 (+ 53us) write_op.cc:312] APPLY: finished
0808 01:10:17.957084 (+ 180us) log.cc:842] Serialized 23 byte log entry
0808 01:10:17.957248 (+ 164us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:10:17.957335 (+ 87us) write_op.cc:454] Released schema lock
0808 01:10:17.957373 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0808 01:10:17.957701 (+ 328us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:10:17.957728 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":12922,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":368493,"apply.run_cpu_time_us":2044,"apply.run_wall_time_us":6618,"num_ops":1,"prepare.queue_time_us":162737,"prepare.run_cpu_time_us":7609,"prepare.run_wall_time_us":439714,"replication_time_us":446607,"thread_start_us":336404,"threads_started":2}]]}
I20260808 01:10:18.082298 1447 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55036
I20260808 01:10:18.082969 1447 raft_consensus.cc:595] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Received request to transfer leadership
I20260808 01:10:18.109520 1516 raft_consensus.cc:1226] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:18.111168 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:18.111544 1603 raft_consensus.cc:1226] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:18.111881 1599 raft_consensus.cc:3032] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:18.112258 1599 raft_consensus.cc:729] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:18.116885 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:18.117662 1525 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:18.112989 1599 consensus_queue.cc:260] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:18.118166 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:18.118402 1599 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:18.113853 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:18.121327 1339 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37234 (request call id 148) took 1338 ms. Trace:
I20260808 01:10:18.125746 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:18.129439 1444 raft_consensus.cc:3032] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:18.121827 1339 rpcz_store.cc:276] 0808 01:10:16.804817 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:16.810884 (+ 6067us) service_pool.cc:227] Handling call
0808 01:10:18.121267 (+1310383us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 01:10:17.183574 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:10:17.184656 (+ 1082us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:10:17.184716 (+ 60us) write_op.cc:435] Acquired schema lock
0808 01:10:17.184759 (+ 43us) tablet.cc:660] Decoding operations
0808 01:10:17.243137 (+ 58378us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:10:17.243223 (+ 86us) write_op.cc:641] Partition lock acquired for write op
0808 01:10:17.243322 (+ 99us) tablet.cc:683] Acquiring locks for 1 operations
0808 01:10:17.243690 (+ 368us) tablet.cc:699] Row locks acquired
0808 01:10:17.243742 (+ 52us) write_op.cc:260] PREPARE: finished
0808 01:10:17.244066 (+ 324us) write_op.cc:270] Start()
0808 01:10:17.244302 (+ 236us) write_op.cc:276] Timestamp: P: 1786151417243982 usec, L: 0
0808 01:10:17.244352 (+ 50us) op_driver.cc:331] REPLICATION: starting
0808 01:10:17.245439 (+ 1087us) log.cc:842] Serialized 536 byte log entry
0808 01:10:17.856410 (+610971us) op_driver.cc:444] REPLICATION: finished
0808 01:10:17.958086 (+101676us) write_op.cc:301] APPLY: starting
0808 01:10:17.958155 (+ 69us) tablet.cc:1365] starting BulkCheckPresence
0808 01:10:17.958249 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0808 01:10:17.958379 (+ 130us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 01:10:17.958558 (+ 179us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 01:10:17.958699 (+ 141us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 01:10:17.958742 (+ 43us) write_op.cc:312] APPLY: finished
0808 01:10:17.958904 (+ 162us) log.cc:842] Serialized 23 byte log entry
0808 01:10:17.959044 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:10:17.959119 (+ 75us) write_op.cc:454] Released schema lock
0808 01:10:17.959149 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0808 01:10:17.959389 (+ 240us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:10:17.959415 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":103066,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":73757,"apply.run_cpu_time_us":1513,"apply.run_wall_time_us":1512,"num_ops":1,"prepare.queue_time_us":213014,"prepare.run_cpu_time_us":4932,"prepare.run_wall_time_us":62212,"replication_time_us":606417,"thread_start_us":66540,"threads_started":1}]]}
I20260808 01:10:18.129663 1444 raft_consensus.cc:729] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:18.128235 2977 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:18.136679 1444 consensus_queue.cc:260] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:18.121976 2971 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:18.137872 2977 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:18.075605 2961 consensus_queue.cc:237] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:18.143333 1594 raft_consensus.cc:1261] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:18.251116 1444 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:10:18.232290 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 63162.5us
I20260808 01:10:18.256269 1340 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55758 (request call id 235) took 2168 ms. Trace:
I20260808 01:10:18.255162 1599 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:18.258818 1447 raft_consensus.cc:3032] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:18.259153 1447 raft_consensus.cc:729] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:18.259800 1447 consensus_queue.cc:260] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:18.261045 1447 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:18.267647 1447 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:18.207010 1342 catalog_manager.cc:5495] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 4 to 5, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 5 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } }
I20260808 01:10:18.256719 1340 rpcz_store.cc:276] 0808 01:10:16.087646 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:16.087789 (+ 143us) service_pool.cc:227] Handling call
0808 01:10:16.211742 (+123953us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b5400002dd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4af0 00007fe34ca47f2a 00007fe352999d6c 00007fe352a93c3e 00007fe350da5fb4 00007fe350da5f42 00007fe350da5e71 00007fe350da5dec 00007fe350da50a2 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:18.256207 (+2044465us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":6,"spinlock_wait_cycles":252810112}
I20260808 01:10:18.176334 1525 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:18.269142 1342 catalog_manager.cc:5495] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:18.245728 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:18.270551 1443 raft_consensus.cc:3032] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:18.270886 1443 raft_consensus.cc:729] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:18.271627 1443 consensus_queue.cc:260] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:18.272873 1443 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:10:18.166390 1390 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 82bb58b6f7c14c22814cfac6d2b2bf6d already in progress
W20260808 01:10:18.195772 2957 log.cc:924] Time spent T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260808 01:10:18.061265 2914 rpcz_store.cc:276] 0808 01:10:15.345276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:15.350639 (+ 5363us) service_pool.cc:227] Handling call
0808 01:10:15.799100 (+448461us) ts_tablet_manager.cc:928] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
0808 01:10:16.097656 (+298556us) spinlock_profiling.cc:243] Waited 1.85 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe349af95fe 00007fe352930a39 00007fe349b168bb 00007fe349b1362a 00007fe349b13552 00007fe351930691
0808 01:10:17.558295 (+1460639us) tablet_copy_client.cc:361] Tablet copy session begun
0808 01:10:17.989139 (+430844us) tablet_metadata.cc:641] Metadata flushed
0808 01:10:17.989456 (+ 317us) tablet_copy_client.cc:490] Wrote new tablet metadata
0808 01:10:18.050114 (+ 60658us) tablet_copy_client.cc:411] Wrote new consensus metadata
0808 01:10:18.050506 (+ 392us) tablet_replica.cc:205] Creating consensus instance
0808 01:10:18.050928 (+ 422us) ts_tablet_manager.cc:983] Replica registered.
0808 01:10:18.060721 (+ 9793us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":37459,"spinlock_wait_cycles":4253952,"tablet-copy.queue_time_us":173270,"thread_start_us":892857,"threads_started":2}
I20260808 01:10:18.275688 2967 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:10:18.274313 1390 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 82bb58b6f7c14c22814cfac6d2b2bf6d already in progress
I20260808 01:10:18.251385 2897 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:18.300801 1443 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:18.293736 1438 raft_consensus.cc:1261] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:18.302968 1520 raft_consensus.cc:1226] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:18.294557 2914 tablet_copy_client.cc:791] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:18.322681 1518 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:18.323863 2914 tablet_copy_client.cc:655] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:18.325815 2895 tablet_bootstrap.cc:492] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:18.326092 1444 raft_consensus.cc:2454] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:18.288937 2967 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:18.343572 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:18.300191 2897 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:18.362360 1410 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:18.371198 2928 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:18.371798 2928 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:18.368402 2970 raft_consensus.cc:2726] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:18.375804 2928 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:18.377585 2967 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:18.379096 1410 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:18.388958 2973 consensus_queue.cc:1035] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:18.398627 2914 tablet_copy_client.cc:531] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:18.400914 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:18.396512 2977 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:18.408905 2877 raft_consensus.cc:2781] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:18.409497 2877 raft_consensus.cc:686] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:18.419061 2897 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:18.420614 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:18.425487 1410 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:18.430421 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:18.432730 2685 raft_consensus.cc:2726] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:18.433701 1490 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:18.442729 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:18.446099 2914 tablet_bootstrap.cc:492] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:18.410420 2877 consensus_queue.cc:237] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:18.448197 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:18.451051 2898 tablet_bootstrap.cc:492] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:18.453790 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:18.454078 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:18.457901 1594 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:18.484107 1597 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:18.501142 1597 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:18.491448 2971 raft_consensus.cc:2781] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:18.560146 2971 raft_consensus.cc:686] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:18.563870 2971 consensus_queue.cc:237] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:10:18.565867 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 90977.3us
I20260808 01:10:18.573568 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:10:18.579882 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 148861us
I20260808 01:10:18.578531 2898 tablet_bootstrap.cc:492] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:18.594156 1597 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:18.594866 1597 raft_consensus.cc:595] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:18.597386 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:18.577728 2874 consensus_queue.cc:1035] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260808 01:10:18.625634 1489 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:18.621939 2898 ts_tablet_manager.cc:1397] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.836s user 0.020s sys 0.036s
I20260808 01:10:18.646070 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:10:18.648365 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 51730.4us
I20260808 01:10:18.652554 1446 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:18.653151 1446 raft_consensus.cc:595] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:10:18.649233 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 50197.1us
I20260808 01:10:18.657100 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:18.657965 1445 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:18.667940 1489 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:18.657829 2792 raft_consensus.cc:2726] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:18.676862 1568 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:18.679876 1489 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:18.686120 2890 raft_consensus.cc:979] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:18.686748 2890 raft_consensus.cc:1067] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
W20260808 01:10:18.705751 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 49274.5us
I20260808 01:10:18.707134 1410 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:18.696113 2874 raft_consensus.cc:2726] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:18.714902 1443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55040
I20260808 01:10:18.715574 1443 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:18.716045 1443 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:18.701344 2995 raft_consensus.cc:2726] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:10:18.726082 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 122275us
I20260808 01:10:18.713335 2838 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:18.738899 1443 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:18.724050 2871 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:18.749326 2871 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:18.749820 2871 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:18.759716 2898 raft_consensus.cc:348] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:18.760707 2898 raft_consensus.cc:729] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:18.762100 2838 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:18.762581 2838 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:10:18.766223 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 60309.5us
I20260808 01:10:18.788581 1446 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:18.789269 1446 raft_consensus.cc:595] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:18.789856 1443 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:18.811400 1603 raft_consensus.cc:1261] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:18.832418 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:18.833278 2207 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:18.838116 1596 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:18.861224 2871 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:18.864682 2871 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:18.867451 2207 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:18.872788 1408 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
W20260808 01:10:18.842774 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 218446us
I20260808 01:10:18.809624 2898 consensus_queue.cc:260] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:10:18.885540 2999 log.cc:924] Time spent T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260808 01:10:18.886741 2898 ts_tablet_manager.cc:1428] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.230s user 0.007s sys 0.001s
W20260808 01:10:18.901782 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 183177us
I20260808 01:10:18.882988 2980 raft_consensus.cc:2781] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:18.904649 1534 tablet_copy_service.cc:342] P 6b09dbefd24b4d52b385c4dff5697dc5: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-7a2611aecd26485e8912ab55890ec0fa received from {username='slave'} at 127.0.0.1:36560
I20260808 01:10:18.908403 1534 tablet_copy_service.cc:433] P 6b09dbefd24b4d52b385c4dff5697dc5: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-7a2611aecd26485e8912ab55890ec0fa on tablet 7a2611aecd26485e8912ab55890ec0fa with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:18.920971 2980 raft_consensus.cc:686] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:18.922101 2980 consensus_queue.cc:237] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:18.872545 2929 consensus_queue.cc:1035] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260808 01:10:18.925743 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:18.926448 1599 raft_consensus.cc:3032] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:18.926795 1599 raft_consensus.cc:729] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:18.927629 1599 consensus_queue.cc:260] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:18.934399 1446 raft_consensus.cc:1226] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 01:10:18.935529 1442 raft_consensus.cc:1261] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:18.937536 1442 raft_consensus.cc:1261] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 01:10:18.941535 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:18.948675 2206 raft_consensus.cc:1203] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:18.971988 1599 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:10:19.010175 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 63197.7us
I20260808 01:10:19.012908 2208 raft_consensus.cc:1226] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:19.018533 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:19.019701 1599 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:18.962903 1338 catalog_manager.cc:5495] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 2 to 3, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:19.025983 1338 catalog_manager.cc:5495] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 2 to 3, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:19.028571 1338 catalog_manager.cc:5495] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 1 to 2, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 2 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:19.056849 2208 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Advancing to term 3
I20260808 01:10:19.074389 2871 raft_consensus.cc:979] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:19.033529 2792 consensus_queue.cc:1035] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260808 01:10:19.027369 2994 consensus_queue.cc:1046] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260808 01:10:19.074975 2871 raft_consensus.cc:1067] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:19.079149 1594 raft_consensus.cc:1261] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:19.081049 1601 raft_consensus.cc:1261] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 01:10:19.083235 2838 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:19.036620 2961 consensus_queue.cc:1035] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260808 01:10:19.097134 1410 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:19.101544 2931 raft_consensus.cc:2726] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:10:19.120491 1596 raft_consensus.cc:595] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:19.120931 1596 raft_consensus.cc:603] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:10:19.121627 1598 raft_consensus.cc:1226] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 2. Current term is 3. Ops: []
W20260808 01:10:19.129448 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 67036.5us
I20260808 01:10:19.142961 2208 raft_consensus.cc:1261] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 01:10:19.118103 2982 consensus_queue.cc:1035] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260808 01:10:19.169319 3002 consensus_queue.cc:1046] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:19.185776 2838 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:19.144335 2973 consensus_queue.cc:1035] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20260808 01:10:19.193022 1600 raft_consensus.cc:1261] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:19.204140 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:19.208990 1445 raft_consensus.cc:2379] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 3: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:19.193419 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:19.199460 1518 raft_consensus.cc:1261] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:19.202529 1522 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58266
I20260808 01:10:19.212409 1522 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:19.212824 1522 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:19.221361 1568 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:19.221833 1522 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:19.225625 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:19.226601 2908 raft_consensus.cc:2726] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:10:19.227329 1522 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:19.228205 1443 raft_consensus.cc:3032] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:19.229066 1443 raft_consensus.cc:729] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:19.225255 2995 consensus_queue.cc:1035] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260808 01:10:19.230201 1443 consensus_queue.cc:260] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:19.234723 1443 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:19.247789 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260808 01:10:19.248764 1443 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:19.276575 1490 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
W20260808 01:10:19.272215 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 64352us
I20260808 01:10:19.285919 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:19.286974 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:19.287595 1602 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:19.299212 2685 consensus_queue.cc:1035] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:19.300590 3015 raft_consensus.cc:2781] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:19.300650 2967 raft_consensus.cc:3032] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:19.314174 1438 raft_consensus.cc:1261] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:19.316087 2967 raft_consensus.cc:729] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:19.320760 2967 consensus_queue.cc:260] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:19.292083 2971 consensus_queue.cc:1035] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260808 01:10:19.334216 2895 tablet_bootstrap.cc:492] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:19.335450 2895 tablet_bootstrap.cc:492] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:19.336479 2895 ts_tablet_manager.cc:1397] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 1.011s user 0.030s sys 0.033s
I20260808 01:10:19.349010 2871 raft_consensus.cc:979] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:19.349611 2871 raft_consensus.cc:1067] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:19.366456 3015 raft_consensus.cc:686] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:19.367455 3015 consensus_queue.cc:237] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:10:19.374156 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 49088.9us
I20260808 01:10:19.377107 1522 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58266
I20260808 01:10:19.377722 1522 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:19.378127 1522 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:10:19.379235 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 54078.3us
I20260808 01:10:19.380980 2961 consensus_queue.cc:1035] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:19.386897 2967 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:19.286875 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:19.413586 1522 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:19.415125 1602 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:19.427480 1522 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:19.419003 1339 catalog_manager.cc:5495] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:19.436560 1339 catalog_manager.cc:5495] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:19.442560 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:19.443421 1602 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:19.460791 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:19.461460 1444 raft_consensus.cc:3032] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:19.461773 1444 raft_consensus.cc:729] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:19.462443 1444 consensus_queue.cc:260] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:19.463488 1444 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:19.467340 1438 raft_consensus.cc:1226] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 01:10:19.481992 1602 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:19.492666 1489 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:19.498706 1444 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:19.496438 2914 tablet_bootstrap.cc:492] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:19.506294 1599 raft_consensus.cc:1261] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:19.499971 2929 raft_consensus.cc:2781] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election won for term 3
W20260808 01:10:19.602121 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 64294.1us
I20260808 01:10:19.607383 2914 tablet_bootstrap.cc:492] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:19.553675 2929 raft_consensus.cc:686] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
W20260808 01:10:19.614105 3021 log.cc:924] Time spent T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260808 01:10:19.552071 3024 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
W20260808 01:10:19.606851 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 50863.4us
I20260808 01:10:19.634441 2914 ts_tablet_manager.cc:1397] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 1.186s user 0.032s sys 0.049s
I20260808 01:10:19.615582 2895 raft_consensus.cc:348] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:19.671998 2895 raft_consensus.cc:729] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:19.672821 2895 consensus_queue.cc:260] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:19.662211 2929 consensus_queue.cc:237] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:10:19.688199 1408 consensus_peers.cc:487] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:10:19.693717 2207 raft_consensus.cc:1261] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:19.695240 2931 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:19.710139 2895 ts_tablet_manager.cc:1428] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.373s user 0.006s sys 0.002s
I20260808 01:10:19.710860 2914 raft_consensus.cc:348] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:19.711977 2914 raft_consensus.cc:729] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:19.712822 2914 consensus_queue.cc:260] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:19.715883 2994 raft_consensus.cc:1050] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:19.731541 1458 tablet_copy_service.cc:342] P 3fb1a6380d27401a831cfe5f86a4cc9d: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-4e0a7f6aa072473098cb2c51da278a83 received from {username='slave'} at 127.0.0.1:60392
I20260808 01:10:19.732026 1458 tablet_copy_service.cc:433] P 3fb1a6380d27401a831cfe5f86a4cc9d: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-4e0a7f6aa072473098cb2c51da278a83 on tablet 4e0a7f6aa072473098cb2c51da278a83 with peer 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:10:19.733624 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 60333.4us
I20260808 01:10:19.734575 2914 ts_tablet_manager.cc:1428] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.090s user 0.004s sys 0.004s
I20260808 01:10:19.736239 1458 tablet_copy_service.cc:342] P 3fb1a6380d27401a831cfe5f86a4cc9d: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-e354fdfbf2264912b39cd20be2cb19f8 received from {username='slave'} at 127.0.0.1:60392
I20260808 01:10:19.736663 1458 tablet_copy_service.cc:433] P 3fb1a6380d27401a831cfe5f86a4cc9d: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-e354fdfbf2264912b39cd20be2cb19f8 on tablet e354fdfbf2264912b39cd20be2cb19f8 with peer 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:10:19.750379 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 60306.4us
W20260808 01:10:19.933216 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 176503us
W20260808 01:10:20.080587 3037 log.cc:924] Time spent T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.293s user 0.001s sys 0.000s
I20260808 01:10:20.116936 2961 raft_consensus.cc:1050] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:20.119419 2961 consensus_queue.cc:237] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:20.120857 2994 consensus_queue.cc:237] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:20.221467 2200 raft_consensus.cc:1203] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20260808 01:10:20.224068 1599 raft_consensus.cc:1261] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:20.233431 2201 raft_consensus.cc:1261] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:20.247318 2208 raft_consensus.cc:1203] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:20.264360 2993 consensus_queue.cc:1035] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260808 01:10:20.276542 1602 raft_consensus.cc:1261] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:10:20.228896 1340 catalog_manager.cc:5495] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 1 to 2, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 2 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:10:20.285604 1597 raft_consensus.cc:1261] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260808 01:10:20.305621 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 342729us
I20260808 01:10:20.438959 3000 consensus_queue.cc:1035] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:20.479187 2202 raft_consensus.cc:1261] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:10:20.501727 2929 consensus_queue.cc:1035] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:20.507599 1340 catalog_manager.cc:5495] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 1 to 2, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 2 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:20.519114 1602 raft_consensus.cc:1261] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:20.523329 2993 consensus_queue.cc:1035] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:20.524654 3012 consensus_queue.cc:1035] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260808 01:10:20.613020 3012 consensus_queue.cc:1035] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:20.656136 3000 raft_consensus.cc:2932] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:20.712299 3054 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:20.713126 3054 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:20.717764 2685 raft_consensus.cc:1050] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:20.710328 1596 raft_consensus.cc:2932] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:20.726699 2931 raft_consensus.cc:1050] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:20.729748 2931 consensus_queue.cc:237] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:20.745272 2685 consensus_queue.cc:237] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:20.763319 2980 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:20.764218 2980 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:20.766656 2980 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:20.753644 3054 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:20.772931 1598 raft_consensus.cc:2932] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:20.773070 3048 raft_consensus.cc:2932] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:20.800086 1446 raft_consensus.cc:1261] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:20.801641 1444 raft_consensus.cc:1261] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:10:20.809414 1444 raft_consensus.cc:1261] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:20.812837 1446 raft_consensus.cc:1261] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:20.855393 2209 raft_consensus.cc:2932] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:20.871184 2207 raft_consensus.cc:2932] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:20.898339 2206 raft_consensus.cc:1261] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:20.900389 1340 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55756 (request call id 243) took 1195 ms. Trace:
I20260808 01:10:20.904783 2209 raft_consensus.cc:1261] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:20.912158 2980 consensus_queue.cc:1035] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:20.929689 2685 consensus_queue.cc:1035] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:20.900893 1340 rpcz_store.cc:276] 0808 01:10:19.705099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:19.705302 (+ 203us) service_pool.cc:227] Handling call
0808 01:10:20.900306 (+1195004us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 01:10:20.708790 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:10:20.728102 (+ 19312us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:10:20.729518 (+ 1416us) write_op.cc:435] Acquired schema lock
0808 01:10:20.729555 (+ 37us) tablet.cc:660] Decoding operations
0808 01:10:20.730125 (+ 570us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:10:20.730247 (+ 122us) write_op.cc:641] Partition lock acquired for write op
0808 01:10:20.730294 (+ 47us) tablet.cc:683] Acquiring locks for 2 operations
0808 01:10:20.730795 (+ 501us) tablet.cc:699] Row locks acquired
0808 01:10:20.730840 (+ 45us) write_op.cc:260] PREPARE: finished
0808 01:10:20.731017 (+ 177us) write_op.cc:270] Start()
0808 01:10:20.731180 (+ 163us) write_op.cc:276] Timestamp: P: 1786151420730957 usec, L: 0
0808 01:10:20.731215 (+ 35us) op_driver.cc:331] REPLICATION: starting
0808 01:10:20.761813 (+ 30598us) log.cc:842] Serialized 929 byte log entry
0808 01:10:20.863110 (+101297us) op_driver.cc:444] REPLICATION: finished
0808 01:10:20.863541 (+ 431us) write_op.cc:301] APPLY: starting
0808 01:10:20.863636 (+ 95us) tablet.cc:1365] starting BulkCheckPresence
0808 01:10:20.863770 (+ 134us) tablet.cc:1368] finished BulkCheckPresence
0808 01:10:20.863935 (+ 165us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 01:10:20.864458 (+ 523us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 01:10:20.864777 (+ 319us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0808 01:10:20.864859 (+ 82us) write_op.cc:312] APPLY: finished
0808 01:10:20.865243 (+ 384us) log.cc:842] Serialized 29 byte log entry
0808 01:10:20.865446 (+ 203us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:10:20.865566 (+ 120us) write_op.cc:454] Released schema lock
0808 01:10:20.865637 (+ 71us) write_op.cc:341] FINISH: Updating metrics
0808 01:10:20.865963 (+ 326us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:10:20.865987 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":142,"apply.run_cpu_time_us":2742,"apply.run_wall_time_us":2741,"num_ops":2,"prepare.queue_time_us":160893,"prepare.run_cpu_time_us":4132,"prepare.run_wall_time_us":59236,"replication_time_us":131791,"thread_start_us":70017,"threads_started":1}]]}
W20260808 01:10:20.946666 3033 log.cc:924] Time spent T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.055s user 0.007s sys 0.008s
I20260808 01:10:20.949066 1338 catalog_manager.cc:5495] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 2 to 5, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:20.961673 3074 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:20.967553 2982 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:20.968228 2982 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:20.970278 2982 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:20.962245 3074 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:20.973927 3074 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:20.985342 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260808 01:10:20.988396 3067 log.cc:924] Time spent T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20260808 01:10:21.006808 3053 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:21.007452 3053 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:21.009838 3053 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:21.010864 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:21.012518 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:21.013652 1338 catalog_manager.cc:5495] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.015246 1567 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:21.020005 3042 raft_consensus.cc:2726] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:10:21.040460 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 71791.4us
I20260808 01:10:21.059479 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:21.059517 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:21.065528 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:21.065691 3016 consensus_queue.cc:1035] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:21.065996 3051 consensus_queue.cc:1035] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:21.066653 1518 raft_consensus.cc:2396] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:10:21.067230 3016 consensus_queue.cc:1035] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:21.068040 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:21.069244 1601 raft_consensus.cc:2396] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:10:21.070283 1338 catalog_manager.cc:5495] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 2 to 3, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } }
I20260808 01:10:21.073159 2971 consensus_queue.cc:1035] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:21.080524 1601 raft_consensus.cc:1261] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:21.114252 1601 raft_consensus.cc:1261] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:21.117067 1518 raft_consensus.cc:1261] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:21.117180 2899 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:21.119330 2899 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:21.123359 2899 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:21.125049 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:21.128208 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:21.130520 2171 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:21.132097 3036 raft_consensus.cc:2726] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:21.132452 1520 consensus_queue.cc:237] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:21.142378 1522 raft_consensus.cc:1261] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:21.143019 1600 raft_consensus.cc:1261] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:21.148221 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:21.143687 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:21.146106 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:21.145108 2971 consensus_queue.cc:1035] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260808 01:10:21.164551 3048 raft_consensus.cc:2932] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.166292 2980 consensus_queue.cc:1035] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:21.167896 2931 consensus_queue.cc:1035] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:21.177488 1601 raft_consensus.cc:2932] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.187991 1410 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:21.196642 1410 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:21.199471 2685 raft_consensus.cc:2726] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:10:21.207393 2207 raft_consensus.cc:1261] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:21.213692 3076 consensus_queue.cc:1035] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260808 01:10:21.196585 3054 raft_consensus.cc:2726] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:21.231578 1410 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:21.221122 3046 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:21.205173 2980 consensus_queue.cc:1035] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260808 01:10:21.210510 1518 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:21.220191 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:21.258271 2892 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:21.258936 2892 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:21.230520 1446 raft_consensus.cc:2932] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.274962 1410 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:21.275558 2685 raft_consensus.cc:2726] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:21.261485 2892 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:21.279330 1439 raft_consensus.cc:2932] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.284152 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:21.247303 1518 raft_consensus.cc:595] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:21.262187 3046 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:21.293357 2973 consensus_queue.cc:1035] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260808 01:10:21.293795 1517 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:21.278102 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:21.298377 1444 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:21.311992 1567 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:21.320134 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:21.321095 1447 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:21.321610 2207 raft_consensus.cc:2932] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.325881 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 82bb58b6f7c14c22814cfac6d2b2bf6d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:10:21.337796 3080 log.cc:924] Time spent T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260808 01:10:21.332823 2985 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:21.341785 2985 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:21.342171 2985 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:21.344484 1490 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:21.347424 2973 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:21.347963 2973 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:21.348296 2973 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:21.348596 3054 raft_consensus.cc:2726] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 01:10:21.350869 3081 log.cc:924] Time spent T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260808 01:10:21.378691 1519 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:21.379320 1519 raft_consensus.cc:595] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:21.394538 1520 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:21.395247 1520 raft_consensus.cc:595] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:21.397738 2980 raft_consensus.cc:2932] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.412433 1339 catalog_manager.cc:5495] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index 5 to 6, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.416906 3046 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:10:21.431375 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 71776.3us
I20260808 01:10:21.416158 1525 raft_consensus.cc:2932] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.443936 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:21.449324 1518 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a19c1759594d4722abadf576352f9a98"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:21.450032 1518 raft_consensus.cc:595] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:21.456696 3051 raft_consensus.cc:979] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:21.457281 3051 raft_consensus.cc:1067] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Signalling peer a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:21.465037 1601 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:45638
I20260808 01:10:21.465683 1601 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:21.466065 1601 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:21.470736 2971 raft_consensus.cc:979] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:21.471428 2971 raft_consensus.cc:1067] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Signalling peer a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:21.476380 3016 raft_consensus.cc:979] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:21.476855 3016 raft_consensus.cc:1067] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:21.477149 1429 tablet_service.cc:1514] Processing DeleteTablet for tablet 82bb58b6f7c14c22814cfac6d2b2bf6d with delete_type TABLET_DATA_TOMBSTONED (TS 3fb1a6380d27401a831cfe5f86a4cc9d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55036
I20260808 01:10:21.482621 1594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:45638
I20260808 01:10:21.483245 1594 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:21.483628 1594 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:21.385502 2973 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:21.495013 1446 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55042
I20260808 01:10:21.495849 1446 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:21.496292 1446 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:21.518687 1601 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:21.526283 1518 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:21.545982 1518 raft_consensus.cc:595] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:21.546398 1518 raft_consensus.cc:603] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:10:21.551949 1601 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:21.555930 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:21.556635 1443 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:10:21.555967 3087 log.cc:924] Time spent T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.183s user 0.000s sys 0.001s
I20260808 01:10:21.556962 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260808 01:10:21.559001 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:21.561875 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:21.562572 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:21.562525 1518 raft_consensus.cc:3032] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:21.562987 1518 raft_consensus.cc:729] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:21.563799 1518 consensus_queue.cc:260] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:21.564851 2973 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:21.565086 1518 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:21.548679 2985 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:21.567094 1598 raft_consensus.cc:2379] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 4: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:10:21.568164 2985 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:21.569128 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:21.569782 1442 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:21.569936 1443 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:10:21.572269 3091 tablet_replica.cc:330] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d: stopping tablet replica
I20260808 01:10:21.573252 3091 raft_consensus.cc:2229] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:21.574084 1446 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:21.576017 1446 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:21.577733 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:21.578397 1524 raft_consensus.cc:3032] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:21.578681 1524 raft_consensus.cc:729] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:21.577883 2685 raft_consensus.cc:2932] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.560853 1594 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:21.581740 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:21.579406 1524 consensus_queue.cc:260] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:21.582350 1599 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:21.582824 1524 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:21.587555 1568 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:21.588581 2985 raft_consensus.cc:2781] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:21.589062 2985 raft_consensus.cc:686] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:21.589920 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:21.592685 1594 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:21.595885 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:21.596692 1519 raft_consensus.cc:2379] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 4: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:10:21.598963 1599 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:21.600196 3091 raft_consensus.cc:2258] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:21.603461 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:21.607041 1601 raft_consensus.cc:2379] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 4: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:10:21.605839 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:21.609038 1519 raft_consensus.cc:3032] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:21.609316 1519 raft_consensus.cc:729] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:21.617050 1519 consensus_queue.cc:260] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:21.618261 1519 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:21.620424 1568 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:21.628702 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:21.629300 1443 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:10:21.632450 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 45814.8us
I20260808 01:10:21.633597 1410 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:21.628391 3053 raft_consensus.cc:2726] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:10:21.641820 3091 ts_tablet_manager.cc:1905] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:21.645181 2985 consensus_queue.cc:237] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:21.645633 2209 raft_consensus.cc:2932] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.648504 2206 raft_consensus.cc:2932] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.653402 1524 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:21.654814 1338 catalog_manager.cc:5495] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.663170 1443 raft_consensus.cc:2454] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:21.669574 1519 raft_consensus.cc:2454] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:21.673769 1567 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:21.674656 3053 raft_consensus.cc:2781] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:21.675179 3053 raft_consensus.cc:686] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:21.676097 3053 consensus_queue.cc:237] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:10:21.678748 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 45225.2us
I20260808 01:10:21.685729 3076 raft_consensus.cc:2781] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:21.701632 1601 raft_consensus.cc:2932] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.705090 3076 raft_consensus.cc:686] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:21.706347 3076 consensus_queue.cc:237] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:21.708633 1594 raft_consensus.cc:2932] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.699013 1524 raft_consensus.cc:2932] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.744611 1338 catalog_manager.cc:5495] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.805027 1524 consensus_queue.cc:237] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:21.779762 1442 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:21.842742 1339 catalog_manager.cc:5495] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 3 to 4, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 4 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:10:21.834707 1518 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:10:21.857321 2209 raft_consensus.cc:1261] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:21.858889 1490 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:21.864032 2971 consensus_queue.cc:1035] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:21.864403 1443 raft_consensus.cc:1261] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:21.861749 3016 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:21.869920 3016 raft_consensus.cc:686] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:21.870868 3016 consensus_queue.cc:237] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:21.880829 1338 catalog_manager.cc:5495] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:21.892853 2961 raft_consensus.cc:2932] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.906136 2973 consensus_queue.cc:1035] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:21.906158 3012 raft_consensus.cc:979] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:21.897845 1442 consensus_queue.cc:237] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:21.908650 3012 raft_consensus.cc:1067] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:21.914183 3091 ts_tablet_manager.cc:1918] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260808 01:10:21.914772 3091 log.cc:1196] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root/wals/82bb58b6f7c14c22814cfac6d2b2bf6d
I20260808 01:10:21.912298 1439 consensus_queue.cc:237] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:21.923151 1443 raft_consensus.cc:2932] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.925096 2209 raft_consensus.cc:2932] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:21.932489 1447 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a19c1759594d4722abadf576352f9a98"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55042
I20260808 01:10:21.933085 1447 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:21.933416 1447 raft_consensus.cc:3037] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:21.942876 2206 raft_consensus.cc:1261] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:21.916936 1329 catalog_manager.cc:4841] TS 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): tablet 82bb58b6f7c14c22814cfac6d2b2bf6d (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:21.947255 1594 raft_consensus.cc:1261] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:21.928929 1339 catalog_manager.cc:5495] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 2 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:10:21.948207 1447 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:21.953295 3074 consensus_queue.cc:1035] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:21.957443 1447 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:21.962251 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:21.962914 1524 raft_consensus.cc:3032] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:21.963178 1524 raft_consensus.cc:729] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:21.965925 1525 raft_consensus.cc:1261] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:21.966591 1524 consensus_queue.cc:260] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:21.967784 1524 raft_consensus.cc:3037] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:21.969198 2206 raft_consensus.cc:1261] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:21.970191 2685 consensus_queue.cc:1035] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:21.973081 3092 consensus_queue.cc:1035] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:21.974555 1338 catalog_manager.cc:5495] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:21.975546 3076 consensus_queue.cc:1035] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:21.979234 1524 raft_consensus.cc:2454] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:21.979108 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:21.981383 2206 raft_consensus.cc:3037] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:21.984320 1410 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:21.994071 2206 raft_consensus.cc:2454] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:22.003011 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 7a2611aecd26485e8912ab55890ec0fa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:22.010466 1342 catalog_manager.cc:5495] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New cstate: current_term: 1 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:22.024258 3076 raft_consensus.cc:2781] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:22.024925 3076 raft_consensus.cc:686] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:22.025920 3076 consensus_queue.cc:237] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:22.029081 3074 raft_consensus.cc:2932] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:22.031250 2931 raft_consensus.cc:2932] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:22.033197 2206 raft_consensus.cc:2932] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:10:22.035941 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 51703.9us
I20260808 01:10:22.039160 2209 raft_consensus.cc:2932] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:22.056445 1329 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4e0a7f6aa072473098cb2c51da278a83 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:22.062229 1518 raft_consensus.cc:2932] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:22.064842 1601 raft_consensus.cc:2932] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:22.098565 1339 catalog_manager.cc:5495] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index 3 to 4, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:22.109722 1586 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a2611aecd26485e8912ab55890ec0fa with delete_type TABLET_DATA_TOMBSTONED (TS a4862d0938c84fc4bd840e4c17acae78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45612
I20260808 01:10:22.113425 1338 catalog_manager.cc:5495] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:22.120805 1511 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e0a7f6aa072473098cb2c51da278a83 with delete_type TABLET_DATA_TOMBSTONED (TS 6b09dbefd24b4d52b385c4dff5697dc5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58250
I20260808 01:10:22.134397 3101 tablet_replica.cc:330] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78: stopping tablet replica
I20260808 01:10:22.140112 3102 tablet_replica.cc:330] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5: stopping tablet replica
I20260808 01:10:22.142314 3102 raft_consensus.cc:2229] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:22.142949 3101 raft_consensus.cc:2229] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:22.143456 1518 raft_consensus.cc:1261] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:22.148319 3102 raft_consensus.cc:2258] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:22.147770 3101 raft_consensus.cc:2258] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:22.148790 1586 tablet_service.cc:1514] Processing DeleteTablet for tablet e354fdfbf2264912b39cd20be2cb19f8 with delete_type TABLET_DATA_TOMBSTONED (TS a4862d0938c84fc4bd840e4c17acae78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45612
I20260808 01:10:22.151428 2892 consensus_queue.cc:1035] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:22.154569 3102 ts_tablet_manager.cc:1905] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:22.156526 3101 ts_tablet_manager.cc:1905] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:22.157639 1329 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e354fdfbf2264912b39cd20be2cb19f8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:22.179184 3101 ts_tablet_manager.cc:1918] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:22.180250 3101 log.cc:1196] T 7a2611aecd26485e8912ab55890ec0fa P a4862d0938c84fc4bd840e4c17acae78: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-2-root/wals/7a2611aecd26485e8912ab55890ec0fa
I20260808 01:10:22.179184 3102 ts_tablet_manager.cc:1918] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:22.182776 1328 catalog_manager.cc:4841] TS a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): tablet 7a2611aecd26485e8912ab55890ec0fa (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:22.183079 3102 log.cc:1196] T 4e0a7f6aa072473098cb2c51da278a83 P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-1-root/wals/4e0a7f6aa072473098cb2c51da278a83
I20260808 01:10:22.185029 1328 catalog_manager.cc:4841] TS 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): tablet 4e0a7f6aa072473098cb2c51da278a83 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:22.185340 3101 tablet_replica.cc:330] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78: stopping tablet replica
I20260808 01:10:22.186529 1446 raft_consensus.cc:1261] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:22.186551 3101 raft_consensus.cc:2229] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:22.187681 3101 raft_consensus.cc:2258] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:22.188246 2892 consensus_queue.cc:1035] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:22.190917 3101 ts_tablet_manager.cc:1905] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:22.195385 1525 raft_consensus.cc:1261] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:10:22.198828 1601 raft_consensus.cc:1261] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:22.202855 2892 consensus_queue.cc:1035] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:22.216884 3101 ts_tablet_manager.cc:1918] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260808 01:10:22.217640 3101 log.cc:1196] T e354fdfbf2264912b39cd20be2cb19f8 P a4862d0938c84fc4bd840e4c17acae78: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-2-root/wals/e354fdfbf2264912b39cd20be2cb19f8
I20260808 01:10:22.219707 1328 catalog_manager.cc:4841] TS a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): tablet e354fdfbf2264912b39cd20be2cb19f8 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:22.232167 3076 consensus_queue.cc:1035] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:22.261104 1447 raft_consensus.cc:1261] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:10:22.261102 1339 catalog_manager.cc:5495] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:22.270048 1524 raft_consensus.cc:1261] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:22.270360 3084 consensus_queue.cc:1035] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:22.271656 3076 consensus_queue.cc:1035] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:22.319927 1601 raft_consensus.cc:1261] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:10:22.321866 3016 consensus_queue.cc:1035] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:22.369933 1446 raft_consensus.cc:1261] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:10:22.374692 3016 consensus_queue.cc:1035] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:22.554718 1518 raft_consensus.cc:1261] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:22.556183 3076 consensus_queue.cc:1035] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:22.575052 2206 raft_consensus.cc:1261] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:22.582906 3074 consensus_queue.cc:1035] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:22.729152 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:23.723905 3128 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:23.724772 3128 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:23.727015 3128 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:23.730854 2982 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:23.731606 2982 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:23.733824 2982 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:23.738435 2894 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:23.739223 2894 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:23.741580 2894 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:23.744978 3136 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:23.767158 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:23.779861 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:23.780215 3136 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:23.782675 3136 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:23.784135 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:10:23.785753 1373 debug-util.cc:398] Leaking SignalData structure 0x7b080003da00 after lost signal to thread 32596
W20260808 01:10:23.786885 1373 debug-util.cc:398] Leaking SignalData structure 0x7b080003a0e0 after lost signal to thread 1376
W20260808 01:10:23.788233 1373 debug-util.cc:398] Leaking SignalData structure 0x7b08003386e0 after lost signal to thread 1473
W20260808 01:10:23.789469 1373 debug-util.cc:398] Leaking SignalData structure 0x7b080036c4c0 after lost signal to thread 1551
I20260808 01:10:23.792891 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:10:23.791915 1373 debug-util.cc:398] Leaking SignalData structure 0x7b0800105300 after lost signal to thread 1629
W20260808 01:10:23.798584 1373 debug-util.cc:398] Leaking SignalData structure 0x7b080010be00 after lost signal to thread 2233
W20260808 01:10:23.809345 1373 debug-util.cc:398] Leaking SignalData structure 0x7b08003597a0 after lost signal to thread 3042
W20260808 01:10:23.809836 1373 debug-util.cc:398] Leaking SignalData structure 0x7b08002c2c60 after lost signal to thread 3063
I20260808 01:10:23.813572 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:23.815929 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:23.816991 3140 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:23.817516 3140 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:23.821049 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:23.819598 3140 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:23.825757 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:23.829174 1522 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:23.829566 3127 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:23.829862 1522 raft_consensus.cc:595] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:23.830257 3127 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:23.831113 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 342510us
I20260808 01:10:23.838212 3127 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:23.842301 1568 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:23.843696 1568 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:23.844069 3136 raft_consensus.cc:2726] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:10:23.847487 3133 raft_consensus.cc:2726] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:10:23.850095 3136 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:23.851815 3136 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:23.854143 3136 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:23.876031 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:23.881529 3141 raft_consensus.cc:979] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:23.882102 3141 raft_consensus.cc:1067] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:23.894011 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:23.902025 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:23.908126 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:23.910602 1568 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:23.916579 1565 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:23.919775 3128 raft_consensus.cc:2726] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:23.921299 3133 raft_consensus.cc:2726] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:23.930189 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:23.938239 1598 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:23.938954 1598 raft_consensus.cc:595] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:23.936756 1410 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:23.941392 3134 raft_consensus.cc:2726] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:23.942263 1522 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:23.942895 1522 raft_consensus.cc:595] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Received request to transfer leadership to TS 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:23.944044 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260808 01:10:23.945703 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:23.946220 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:23.964176 1410 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:23.967193 3134 raft_consensus.cc:2726] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:23.979655 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:23.979800 1439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55042
I20260808 01:10:23.980926 1439 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:23.981329 1439 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:23.989645 1439 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:23.999004 1439 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:23.999197 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:23.999918 2206 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:24.002970 1490 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:24.004227 3140 raft_consensus.cc:2726] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:10:24.008355 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:24.008929 1522 raft_consensus.cc:3032] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:24.009200 1522 raft_consensus.cc:729] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:24.009997 2206 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:24.010015 1522 consensus_queue.cc:260] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:24.011376 1522 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:24.011492 1408 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:24.012552 3145 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:24.013078 3145 raft_consensus.cc:686] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:24.013967 3145 consensus_queue.cc:237] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:24.019343 1522 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:24.044977 1439 consensus_queue.cc:237] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:24.056519 1598 raft_consensus.cc:1261] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:24.056583 1522 raft_consensus.cc:1261] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:24.058152 3145 consensus_queue.cc:1035] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:24.059221 3134 consensus_queue.cc:1035] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260808 01:10:24.069694 1408 consensus_peers.cc:487] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a668c2efd2dd4f5b946b70904c143c17. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:24.058799 1338 catalog_manager.cc:5495] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:24.090569 3134 raft_consensus.cc:2932] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:24.125499 1598 raft_consensus.cc:2932] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:24.134080 1522 raft_consensus.cc:2932] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:24.136977 2894 raft_consensus.cc:979] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:24.137524 3133 raft_consensus.cc:1067] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:24.144654 1518 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58280
I20260808 01:10:24.145442 1518 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:24.146077 1518 raft_consensus.cc:3037] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:24.172551 1518 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:24.178325 1518 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:24.181336 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:24.181945 2206 raft_consensus.cc:3037] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:24.196820 2206 raft_consensus.cc:2454] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:24.198138 1487 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:24.199193 3126 raft_consensus.cc:2781] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:24.192094 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:24.199690 3126 raft_consensus.cc:686] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:24.199929 1598 raft_consensus.cc:3032] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:24.200256 1598 raft_consensus.cc:729] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:24.200690 3126 consensus_queue.cc:237] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:24.202304 1598 consensus_queue.cc:260] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:24.203617 1598 raft_consensus.cc:3037] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:24.223997 1339 catalog_manager.cc:5495] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:24.226562 1599 consensus_queue.cc:237] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:24.229909 1598 raft_consensus.cc:2454] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:24.249289 1439 raft_consensus.cc:1261] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 01:10:24.249661 1565 consensus_peers.cc:487] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f0008d696f943a3aaa520463af63298. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:24.251760 3128 consensus_queue.cc:1035] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:24.268801 1518 raft_consensus.cc:1261] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:24.270298 3133 consensus_queue.cc:1035] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:24.301965 3135 raft_consensus.cc:979] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:24.302541 3140 raft_consensus.cc:1067] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Signalling peer 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:24.305505 2206 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:49268
I20260808 01:10:24.306077 2206 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:24.306406 2206 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:24.314267 1339 catalog_manager.cc:5495] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 1 to 2, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 2 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:24.323760 3128 raft_consensus.cc:2932] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:24.331393 1518 raft_consensus.cc:2932] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:24.328835 1439 raft_consensus.cc:2932] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:24.336360 2206 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:24.347380 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:24.348193 1522 raft_consensus.cc:3032] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:24.348686 1522 raft_consensus.cc:729] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:24.349579 1522 consensus_queue.cc:260] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:24.350924 1522 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:24.352747 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:24.353554 1599 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:24.370429 1599 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:10:24.371846 2206 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:24.372766 2171 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:24.376139 3125 raft_consensus.cc:2781] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:24.376741 3125 raft_consensus.cc:686] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:24.377681 3125 consensus_queue.cc:237] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:24.403313 1339 catalog_manager.cc:5495] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 3 to 4, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 4 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:24.407225 1522 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:10:24.409369 1342 catalog_manager.cc:5495] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:24.420156 1524 consensus_queue.cc:237] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:24.473639 1599 raft_consensus.cc:1261] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260808 01:10:24.473745 1487 consensus_peers.cc:487] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2355fe5dfc824ec78c044695be9d32de. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:24.476199 3135 consensus_queue.cc:1035] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:24.499800 1439 raft_consensus.cc:1261] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:24.503638 3135 raft_consensus.cc:2932] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:24.521375 2206 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:24.522733 3076 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:24.521656 1599 raft_consensus.cc:2932] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:24.525692 1516 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:24.534569 3134 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:24.538143 1439 consensus_queue.cc:237] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:24.540452 3140 consensus_queue.cc:1035] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:24.530939 1339 catalog_manager.cc:5495] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 2 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:24.568240 1443 raft_consensus.cc:2932] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:24.582791 1601 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:24.585553 1518 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 01:10:24.586887 1408 consensus_peers.cc:487] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 516db2422f0644aebfd1999735efdbb8. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:24.587044 3145 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:24.599625 3145 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:24.700834 3076 raft_consensus.cc:2932] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:24.712167 1524 raft_consensus.cc:2932] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:24.725317 1599 raft_consensus.cc:1261] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260808 01:10:24.730477 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 100693us
I20260808 01:10:24.782996 3172 consensus_queue.cc:1035] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
W20260808 01:10:24.812784 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 59866.1us
I20260808 01:10:24.820766 1601 raft_consensus.cc:2932] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:24.831172 3177 ts_tablet_manager.cc:927] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
W20260808 01:10:24.876489 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 69028us
I20260808 01:10:24.912705 1599 raft_consensus.cc:1261] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:10:24.914407 2201 raft_consensus.cc:1261] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:24.914716 3185 ts_tablet_manager.cc:927] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:24.916502 1340 catalog_manager.cc:5495] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:24.928251 2899 consensus_queue.cc:1035] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260808 01:10:24.928980 3184 consensus_queue.cc:1035] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260808 01:10:24.964160 1517 raft_consensus.cc:1261] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:10:24.974021 3125 consensus_queue.cc:1035] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260808 01:10:24.969717 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 72803.2us
I20260808 01:10:24.995067 3177 tablet_copy_client.cc:316] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.65:38291
I20260808 01:10:25.053902 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:25.057652 1458 tablet_copy_service.cc:140] P 3fb1a6380d27401a831cfe5f86a4cc9d: Received BeginTabletCopySession request for tablet a668c2efd2dd4f5b946b70904c143c17 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:60392)
I20260808 01:10:25.058326 1458 tablet_copy_service.cc:161] P 3fb1a6380d27401a831cfe5f86a4cc9d: Beginning new tablet copy session on tablet a668c2efd2dd4f5b946b70904c143c17 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:60392: session id = 95b661b03f1d460b9657e61d4b0247cd-a668c2efd2dd4f5b946b70904c143c17
I20260808 01:10:25.045953 3185 tablet_copy_client.cc:316] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.67:35675
I20260808 01:10:25.106148 1458 tablet_copy_source_session.cc:205] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:25.106508 3199 ts_tablet_manager.cc:927] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:25.135406 1613 tablet_copy_service.cc:140] P a4862d0938c84fc4bd840e4c17acae78: Received BeginTabletCopySession request for tablet 4f0008d696f943a3aaa520463af63298 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:34182)
I20260808 01:10:25.153419 1613 tablet_copy_service.cc:161] P a4862d0938c84fc4bd840e4c17acae78: Beginning new tablet copy session on tablet 4f0008d696f943a3aaa520463af63298 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:34182: session id = 95b661b03f1d460b9657e61d4b0247cd-4f0008d696f943a3aaa520463af63298
I20260808 01:10:25.177542 3199 tablet_copy_client.cc:316] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.66:37093
I20260808 01:10:25.179297 1446 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:25.180107 1446 raft_consensus.cc:595] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:25.185977 3177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a668c2efd2dd4f5b946b70904c143c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:25.208317 1534 tablet_copy_service.cc:140] P 6b09dbefd24b4d52b385c4dff5697dc5: Received BeginTabletCopySession request for tablet 2355fe5dfc824ec78c044695be9d32de from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:36560)
I20260808 01:10:25.234115 1534 tablet_copy_service.cc:161] P 6b09dbefd24b4d52b385c4dff5697dc5: Beginning new tablet copy session on tablet 2355fe5dfc824ec78c044695be9d32de from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:36560: session id = 95b661b03f1d460b9657e61d4b0247cd-2355fe5dfc824ec78c044695be9d32de
I20260808 01:10:25.241891 3204 ts_tablet_manager.cc:927] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
I20260808 01:10:25.246122 1524 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:25.246783 1524 raft_consensus.cc:595] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:25.274713 1522 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:25.275329 1522 raft_consensus.cc:595] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:25.278753 1613 tablet_copy_source_session.cc:205] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78: Tablet Copy: opened 0 blocks and 1 log segments
W20260808 01:10:25.281134 3198 log.cc:924] Time spent T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.070s user 0.003s sys 0.000s
I20260808 01:10:25.282546 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260808 01:10:25.311307 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:25.285279 1534 tablet_copy_source_session.cc:205] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:25.311981 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:25.337783 3199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2355fe5dfc824ec78c044695be9d32de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:25.389582 3185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0008d696f943a3aaa520463af63298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:25.440055 3204 tablet_copy_client.cc:316] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.65:38291
I20260808 01:10:25.414116 3184 raft_consensus.cc:979] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:25.443544 1458 tablet_copy_service.cc:140] P 3fb1a6380d27401a831cfe5f86a4cc9d: Received BeginTabletCopySession request for tablet 516db2422f0644aebfd1999735efdbb8 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:60392)
I20260808 01:10:25.444211 1458 tablet_copy_service.cc:161] P 3fb1a6380d27401a831cfe5f86a4cc9d: Beginning new tablet copy session on tablet 516db2422f0644aebfd1999735efdbb8 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:60392: session id = 95b661b03f1d460b9657e61d4b0247cd-516db2422f0644aebfd1999735efdbb8
I20260808 01:10:25.443606 3184 raft_consensus.cc:1067] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Signalling peer a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:25.449308 3177 tablet_copy_client.cc:791] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:25.452291 3177 tablet_copy_client.cc:655] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:25.459692 3177 tablet_copy_client.cc:531] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:25.466004 1594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:45638
I20260808 01:10:25.466640 1594 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:25.467335 1594 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:25.475116 1594 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:25.475543 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 53341.3us
I20260808 01:10:25.493525 1594 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:25.482416 1458 tablet_copy_source_session.cc:205] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:25.494663 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:25.501667 1516 raft_consensus.cc:3032] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:25.501991 1516 raft_consensus.cc:729] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:25.502714 1516 consensus_queue.cc:260] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:25.504904 1516 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:25.507061 3199 tablet_copy_client.cc:791] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:25.497316 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:25.507737 3199 tablet_copy_client.cc:655] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:25.508328 2209 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:25.510568 3177 tablet_bootstrap.cc:492] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:25.520741 2209 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:25.522113 1565 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:25.522778 3204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 516db2422f0644aebfd1999735efdbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:25.523016 3175 raft_consensus.cc:2781] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:25.524169 3175 raft_consensus.cc:686] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:25.525022 3175 consensus_queue.cc:237] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:25.537753 1516 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:25.539243 3199 tablet_copy_client.cc:531] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:25.549531 3185 tablet_copy_client.cc:791] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:25.552381 3185 tablet_copy_client.cc:655] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:25.565627 3185 tablet_copy_client.cc:531] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:25.571133 3199 tablet_bootstrap.cc:492] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:25.591308 1342 catalog_manager.cc:5495] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 2 to 3, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 3 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:25.601490 3185 tablet_bootstrap.cc:492] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:25.654855 3174 raft_consensus.cc:979] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:25.655622 3174 raft_consensus.cc:1067] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:25.671437 1446 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55042
I20260808 01:10:25.672447 1446 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:25.672855 1446 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:25.686115 1446 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:25.688455 1446 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:25.689996 3183 raft_consensus.cc:979] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:25.693332 3183 raft_consensus.cc:1067] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Signalling peer a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:25.695366 1594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:45628
I20260808 01:10:25.695983 1594 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:25.696360 1594 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:25.698096 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:25.698858 1604 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:25.704249 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:25.705233 1516 raft_consensus.cc:3032] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:10:25.705535 1516 raft_consensus.cc:729] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:25.706228 1516 consensus_queue.cc:260] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:25.707301 1516 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:25.680042 3204 tablet_copy_client.cc:791] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:25.726708 3204 tablet_copy_client.cc:655] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:25.746934 3204 tablet_copy_client.cc:531] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:25.750622 1516 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:10:25.751184 1604 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:10:25.755158 1594 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:25.758966 1594 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:25.764524 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:25.764974 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:25.765210 1410 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:25.765211 1516 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:25.769851 3145 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:10:25.771672 3145 raft_consensus.cc:686] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:25.772200 1439 raft_consensus.cc:3032] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:25.774348 1439 raft_consensus.cc:729] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:25.776329 1516 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:25.781390 1567 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:25.780928 1439 consensus_queue.cc:260] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:25.784529 3133 raft_consensus.cc:2781] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:25.784381 1439 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:25.788156 3133 raft_consensus.cc:686] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:25.789186 3133 consensus_queue.cc:237] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:25.798405 1439 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:25.805892 3185 tablet_bootstrap.cc:492] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:25.810101 3185 tablet_bootstrap.cc:492] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:25.813207 3185 ts_tablet_manager.cc:1397] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.212s user 0.039s sys 0.010s
I20260808 01:10:25.779664 3145 consensus_queue.cc:237] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:25.872504 3185 raft_consensus.cc:348] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:25.878773 3185 raft_consensus.cc:729] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:25.881572 3185 consensus_queue.cc:260] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:25.904738 3185 ts_tablet_manager.cc:1428] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.091s user 0.003s sys 0.009s
I20260808 01:10:25.925968 3204 tablet_bootstrap.cc:492] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:25.939250 3177 tablet_bootstrap.cc:492] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260808 01:10:25.940133 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 49802.2us
I20260808 01:10:25.956923 1342 catalog_manager.cc:5495] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 2 to 3, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 3 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:10:25.951930 1613 tablet_copy_service.cc:342] P a4862d0938c84fc4bd840e4c17acae78: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-4f0008d696f943a3aaa520463af63298 received from {username='slave'} at 127.0.0.1:34182
I20260808 01:10:25.961004 3177 tablet_bootstrap.cc:492] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:25.963567 1613 tablet_copy_service.cc:433] P a4862d0938c84fc4bd840e4c17acae78: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-4f0008d696f943a3aaa520463af63298 on tablet 4f0008d696f943a3aaa520463af63298 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:25.994210 3177 ts_tablet_manager.cc:1397] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.484s user 0.034s sys 0.026s
I20260808 01:10:26.008656 1340 catalog_manager.cc:5495] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 4 to 5, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 5 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:26.035571 3199 tablet_bootstrap.cc:492] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:26.037251 3199 tablet_bootstrap.cc:492] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:26.037976 3199 ts_tablet_manager.cc:1397] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.467s user 0.022s sys 0.029s
I20260808 01:10:26.064054 2209 raft_consensus.cc:1203] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:26.058125 3199 raft_consensus.cc:348] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:26.066056 3199 raft_consensus.cc:729] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:26.068264 2209 raft_consensus.cc:1261] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:10:26.066890 3199 consensus_queue.cc:260] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:26.072392 3199 ts_tablet_manager.cc:1428] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.034s user 0.005s sys 0.002s
I20260808 01:10:26.073604 3176 consensus_queue.cc:1035] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:26.081197 1534 tablet_copy_service.cc:342] P 6b09dbefd24b4d52b385c4dff5697dc5: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-2355fe5dfc824ec78c044695be9d32de received from {username='slave'} at 127.0.0.1:36560
I20260808 01:10:26.081615 1534 tablet_copy_service.cc:433] P 6b09dbefd24b4d52b385c4dff5697dc5: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-2355fe5dfc824ec78c044695be9d32de on tablet 2355fe5dfc824ec78c044695be9d32de with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:26.083503 3177 raft_consensus.cc:348] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:26.103577 3177 raft_consensus.cc:729] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:26.104351 3177 consensus_queue.cc:260] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:26.115590 3177 ts_tablet_manager.cc:1428] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.121s user 0.009s sys 0.000s
I20260808 01:10:26.116793 1518 raft_consensus.cc:1261] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:10:26.120324 1458 tablet_copy_service.cc:342] P 3fb1a6380d27401a831cfe5f86a4cc9d: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-a668c2efd2dd4f5b946b70904c143c17 received from {username='slave'} at 127.0.0.1:60392
I20260808 01:10:26.120820 1458 tablet_copy_service.cc:433] P 3fb1a6380d27401a831cfe5f86a4cc9d: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-a668c2efd2dd4f5b946b70904c143c17 on tablet a668c2efd2dd4f5b946b70904c143c17 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:26.122507 3176 consensus_queue.cc:1035] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:26.192121 1445 raft_consensus.cc:1261] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:26.198498 3133 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260808 01:10:26.284639 1517 raft_consensus.cc:1261] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:10:26.331058 1518 raft_consensus.cc:1261] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:26.333168 3219 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260808 01:10:26.353574 3183 consensus_queue.cc:1035] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260808 01:10:26.413936 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:26.413440 2201 raft_consensus.cc:1203] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
W20260808 01:10:26.486680 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 72551.1us
W20260808 01:10:26.514528 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 62536.1us
I20260808 01:10:26.537484 1599 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:26.538161 1599 raft_consensus.cc:595] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:26.588004 3216 raft_consensus.cc:1050] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:26.592067 2201 raft_consensus.cc:1203] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260808 01:10:26.594226 3216 consensus_queue.cc:237] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:26.696244 3204 tablet_bootstrap.cc:492] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:26.697461 3204 tablet_bootstrap.cc:492] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:26.698383 3204 ts_tablet_manager.cc:1397] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.773s user 0.056s sys 0.051s
W20260808 01:10:26.704989 3236 log.cc:924] Time spent T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.226s user 0.004s sys 0.002s
I20260808 01:10:26.712916 1599 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:51424
W20260808 01:10:26.700721 3244 log.cc:924] Time spent T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260808 01:10:26.725150 1599 raft_consensus.cc:595] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
W20260808 01:10:26.738269 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 202480us
W20260808 01:10:26.743882 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 127447us
W20260808 01:10:26.754570 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 71355.1us
I20260808 01:10:26.771641 3204 raft_consensus.cc:348] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:26.779314 1446 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:26.780045 1446 raft_consensus.cc:595] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:26.776894 3204 raft_consensus.cc:729] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:26.789330 3145 raft_consensus.cc:1050] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:26.789904 3204 consensus_queue.cc:260] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:26.792708 3145 consensus_queue.cc:237] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:26.814623 1446 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bc8c41cadc374229baf248366012f202"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:26.815243 1446 raft_consensus.cc:595] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:26.818300 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260808 01:10:26.819291 3175 raft_consensus.cc:979] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:26.819809 3175 raft_consensus.cc:1067] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:26.831003 2201 raft_consensus.cc:1203] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:26.820123 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:26.832381 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:26.842943 1446 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55040
I20260808 01:10:26.845126 1446 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:26.846961 1446 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:26.848212 3204 ts_tablet_manager.cc:1428] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.149s user 0.009s sys 0.002s
I20260808 01:10:26.862012 1439 raft_consensus.cc:1261] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:26.862428 2209 raft_consensus.cc:1261] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:26.870337 1458 tablet_copy_service.cc:342] P 3fb1a6380d27401a831cfe5f86a4cc9d: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-516db2422f0644aebfd1999735efdbb8 received from {username='slave'} at 127.0.0.1:60392
I20260808 01:10:26.870854 1458 tablet_copy_service.cc:433] P 3fb1a6380d27401a831cfe5f86a4cc9d: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-516db2422f0644aebfd1999735efdbb8 on tablet 516db2422f0644aebfd1999735efdbb8 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:26.871001 3241 consensus_queue.cc:1035] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:26.872879 3076 raft_consensus.cc:979] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:26.873375 3076 raft_consensus.cc:1067] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:26.875779 3237 consensus_queue.cc:1035] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:26.908823 1599 raft_consensus.cc:1261] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:10:26.910835 1518 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58266
I20260808 01:10:26.911643 1518 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:26.912043 1518 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:26.912925 1446 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:26.917934 1446 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:26.919080 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:26.920064 2204 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:26.964901 1518 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:27.000628 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:27.004313 1601 raft_consensus.cc:3032] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:27.000838 1518 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:27.007037 1601 raft_consensus.cc:729] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:27.008836 1601 consensus_queue.cc:260] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:27.017920 1601 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:27.021911 1524 raft_consensus.cc:1261] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:27.022954 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:27.025604 1599 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:27.027504 3176 consensus_queue.cc:1035] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260808 01:10:27.028416 3228 consensus_queue.cc:1035] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260808 01:10:27.054387 1601 raft_consensus.cc:2454] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:27.068768 1599 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:27.073125 1489 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:10:27.093403 3255 log.cc:924] Time spent T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.172s user 0.002s sys 0.000s
I20260808 01:10:27.087677 3259 raft_consensus.cc:2781] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:27.106673 3227 raft_consensus.cc:979] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:27.107331 3227 raft_consensus.cc:1067] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:27.109781 3259 raft_consensus.cc:686] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:27.111086 3259 consensus_queue.cc:237] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:27.131911 3184 raft_consensus.cc:1050] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
W20260808 01:10:27.162279 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 112462us
I20260808 01:10:27.136255 3184 consensus_queue.cc:237] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:27.166250 2204 raft_consensus.cc:2454] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:27.196431 1410 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:27.196739 2209 raft_consensus.cc:1261] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:27.211771 1599 raft_consensus.cc:1261] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:27.223591 1516 raft_consensus.cc:1261] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:27.231815 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:27.232707 1445 raft_consensus.cc:3032] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:27.232996 1445 raft_consensus.cc:729] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:27.243664 1516 raft_consensus.cc:1226] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:27.233973 1445 consensus_queue.cc:260] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:27.245357 1445 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:27.233086 3247 raft_consensus.cc:2781] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:27.264343 3247 raft_consensus.cc:686] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:27.267063 3269 consensus_queue.cc:1035] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260808 01:10:27.266132 3247 consensus_queue.cc:237] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:27.275907 1525 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bc8c41cadc374229baf248366012f202"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58266
I20260808 01:10:27.276742 1525 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:27.277199 1525 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:27.301131 3265 consensus_queue.cc:1035] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
W20260808 01:10:27.303256 3272 log.cc:924] Time spent T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.061s user 0.003s sys 0.000s
I20260808 01:10:27.336740 1525 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:10:27.354554 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 70592.8us
I20260808 01:10:27.358964 1525 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:27.360766 3133 raft_consensus.cc:979] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:27.361270 3133 raft_consensus.cc:1067] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:27.361176 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:27.361862 1442 raft_consensus.cc:3032] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:27.362154 1442 raft_consensus.cc:729] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
W20260808 01:10:27.362219 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 80629.7us
I20260808 01:10:27.362890 1442 consensus_queue.cc:260] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:27.367151 1442 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:27.375125 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:27.375989 1601 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:27.379303 1439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55040
I20260808 01:10:27.380065 1439 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:27.380484 1439 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:27.355283 3226 consensus_queue.cc:1035] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260808 01:10:27.384344 2204 raft_consensus.cc:1261] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:27.391243 2209 raft_consensus.cc:2932] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:27.417069 1339 catalog_manager.cc:5495] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 1 to 2, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 2 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:27.408713 1444 raft_consensus.cc:2932] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:10:27.481770 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 76285.3us
W20260808 01:10:27.489045 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 75785.7us
W20260808 01:10:27.489601 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 83839.9us
I20260808 01:10:27.506594 3260 consensus_queue.cc:1035] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20260808 01:10:27.412006 1517 raft_consensus.cc:2932] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:10:27.520105 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 95791.8us
I20260808 01:10:27.529381 1446 raft_consensus.cc:1261] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260808 01:10:27.544332 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 50905.3us
I20260808 01:10:27.551198 1445 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:27.555804 1599 raft_consensus.cc:1261] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:27.556073 1442 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:27.390049 3270 raft_consensus.cc:2932] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:27.565909 3262 consensus_queue.cc:1035] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260808 01:10:27.571683 3140 consensus_queue.cc:1035] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:27.574455 1490 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:27.576821 1439 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:27.581224 1440 raft_consensus.cc:1226] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:27.575455 1601 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:27.595007 1439 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:27.602555 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:27.603444 1525 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:27.605844 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:27.607686 1599 raft_consensus.cc:3032] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:27.608017 1599 raft_consensus.cc:729] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:27.614313 2204 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Advancing to term 2
I20260808 01:10:27.617091 1525 raft_consensus.cc:2454] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:27.624785 1410 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:27.602905 3262 raft_consensus.cc:2781] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:27.631212 1445 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:27.636613 2971 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:27.637086 3234 raft_consensus.cc:2781] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:27.637953 3234 raft_consensus.cc:686] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:27.641012 3234 consensus_queue.cc:237] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:27.609392 3251 consensus_queue.cc:1046] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260808 01:10:27.637218 2204 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:27.649883 3262 raft_consensus.cc:686] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:27.677737 1342 catalog_manager.cc:5495] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:27.688990 3262 consensus_queue.cc:237] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:27.733254 1604 raft_consensus.cc:2932] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:27.733980 3173 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260808 01:10:27.736531 3258 raft_consensus.cc:2932] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:27.753417 1599 consensus_queue.cc:260] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:27.784089 2209 raft_consensus.cc:2932] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:27.761194 3183 raft_consensus.cc:2932] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:27.789724 1604 raft_consensus.cc:2932] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:27.805985 1522 raft_consensus.cc:2932] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:27.816568 1517 raft_consensus.cc:1226] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
W20260808 01:10:27.829627 3278 log.cc:924] Time spent T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
I20260808 01:10:27.845762 1604 raft_consensus.cc:1261] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:27.849061 1601 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:27.857463 3184 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:27.865016 1599 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:27.890365 1599 raft_consensus.cc:2454] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:27.870071 3076 consensus_queue.cc:1035] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260808 01:10:27.914903 2204 raft_consensus.cc:1261] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:27.984292 1340 catalog_manager.cc:5495] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:27.963614 3227 consensus_queue.cc:1035] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260808 01:10:28.060732 3140 raft_consensus.cc:1050] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:28.064288 1444 raft_consensus.cc:2932] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:28.072978 2206 raft_consensus.cc:2932] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:28.075541 1599 consensus_queue.cc:237] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:28.076248 1339 catalog_manager.cc:5495] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:28.092890 1522 raft_consensus.cc:1261] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:28.095615 2204 raft_consensus.cc:1261] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:28.105050 3237 consensus_queue.cc:1035] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:28.104269 3140 consensus_queue.cc:237] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:28.112175 3216 consensus_queue.cc:1035] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:28.165282 3262 raft_consensus.cc:1013] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: attempt to promote peer 95b661b03f1d460b9657e61d4b0247cd: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260808 01:10:28.174659 2204 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:28.180673 1599 raft_consensus.cc:1261] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:28.186400 1522 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58250
I20260808 01:10:28.180138 1342 catalog_manager.cc:5495] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 2 to 3, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:28.187077 1522 raft_consensus.cc:595] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Received request to transfer leadership
I20260808 01:10:28.184867 1439 raft_consensus.cc:1261] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:28.173641 3237 raft_consensus.cc:2932] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:28.161641 2209 raft_consensus.cc:2932] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:28.201144 3172 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260808 01:10:28.208516 1601 raft_consensus.cc:1261] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:28.223382 3140 raft_consensus.cc:979] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:28.219584 1594 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:28.224151 3140 raft_consensus.cc:1067] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Signalling peer 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:28.232167 2204 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:49268
I20260808 01:10:28.233145 2204 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:28.233804 2204 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:28.236037 1439 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:28.238451 3243 consensus_queue.cc:1035] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:28.241595 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:28.252681 3140 consensus_queue.cc:1035] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:28.261255 3289 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:28.265539 1522 raft_consensus.cc:2932] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:28.267295 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4f0008d696f943a3aaa520463af63298 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:28.260937 2204 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:28.277868 1340 catalog_manager.cc:5495] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 2 to 3, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 3 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:28.270526 3184 consensus_queue.cc:1035] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260808 01:10:28.288010 3258 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260808 01:10:28.302182 2204 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:28.303951 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:28.304925 1598 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:28.306090 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:28.306674 1444 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:28.326109 1444 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:10:28.326793 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:28.327554 1525 raft_consensus.cc:3032] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:28.327816 1525 raft_consensus.cc:729] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:28.328562 1440 raft_consensus.cc:1226] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260808 01:10:28.328565 1525 consensus_queue.cc:260] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:28.332198 1525 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:28.331729 1522 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58250
I20260808 01:10:28.338600 1525 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:10:28.339766 2171 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:28.339732 1522 raft_consensus.cc:595] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Received request to transfer leadership
I20260808 01:10:28.342159 1522 raft_consensus.cc:603] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:10:28.339430 1598 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:10:28.344678 1599 raft_consensus.cc:1226] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260808 01:10:28.343811 1338 catalog_manager.cc:5495] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:28.358368 1524 raft_consensus.cc:1261] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:28.361918 3242 raft_consensus.cc:2781] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election won for term 3
W20260808 01:10:28.388283 1390 auto_rebalancer.cc:663] Could not move replica: Illegal state: not currently leader
W20260808 01:10:28.388746 1390 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: not currently leader
I20260808 01:10:28.372419 3267 consensus_queue.cc:1035] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260808 01:10:28.402882 3242 raft_consensus.cc:686] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:28.405315 1604 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:28.406011 1604 raft_consensus.cc:595] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:28.408900 1440 consensus_queue.cc:237] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:28.422313 1524 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f6bb0551ce74437284392981d2921446"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:28.423079 1524 raft_consensus.cc:595] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:28.433151 2207 raft_consensus.cc:1261] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:28.440765 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 01:10:28.443295 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:28.462695 1600 raft_consensus.cc:1261] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:28.465904 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:28.448674 3227 consensus_queue.cc:1035] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260808 01:10:28.469254 3243 consensus_queue.cc:1035] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:28.455690 3242 consensus_queue.cc:237] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:28.481863 1342 catalog_manager.cc:5495] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:10:28.612345 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 113893us
W20260808 01:10:28.613441 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 114585us
I20260808 01:10:28.616252 3183 raft_consensus.cc:2932] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:28.649788 1429 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f0008d696f943a3aaa520463af63298 with delete_type TABLET_DATA_TOMBSTONED (TS 3fb1a6380d27401a831cfe5f86a4cc9d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55036
I20260808 01:10:28.676666 2207 raft_consensus.cc:2932] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:10:28.678815 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 46432.3us
I20260808 01:10:28.684654 1601 raft_consensus.cc:2932] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:28.695394 1329 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a668c2efd2dd4f5b946b70904c143c17 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:10:28.936506 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 152237us
I20260808 01:10:28.946741 3309 tablet_replica.cc:330] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d: stopping tablet replica
I20260808 01:10:28.947959 3309 raft_consensus.cc:2229] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:28.948796 3309 raft_consensus.cc:2258] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:28.951684 1524 consensus_queue.cc:237] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:28.983822 1599 raft_consensus.cc:1261] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:28.992771 3309 ts_tablet_manager.cc:1905] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:29.003048 3259 consensus_queue.cc:1035] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260808 01:10:29.008370 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 123796us
I20260808 01:10:29.015168 1340 catalog_manager.cc:5495] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index 4 to 5, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:29.021545 1338 catalog_manager.cc:5495] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 2 to 3, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68), now has a pending config: VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65), VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66), VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67), VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 3 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:10:29.106878 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 55349.8us
I20260808 01:10:29.116673 1511 tablet_service.cc:1514] Processing DeleteTablet for tablet a668c2efd2dd4f5b946b70904c143c17 with delete_type TABLET_DATA_TOMBSTONED (TS 6b09dbefd24b4d52b385c4dff5697dc5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58250
I20260808 01:10:29.164574 2204 raft_consensus.cc:1261] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:29.181077 3259 consensus_queue.cc:1035] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260808 01:10:29.214285 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 70759.1us
I20260808 01:10:29.216521 1604 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:10:29.245798 3326 raft_consensus.cc:979] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:29.246443 3326 raft_consensus.cc:1067] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:29.240128 3238 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.017s
I20260808 01:10:29.284080 1524 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58280
I20260808 01:10:29.284678 1524 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:29.285061 1524 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:10:29.305035 3336 log.cc:924] Time spent T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260808 01:10:29.308920 1439 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:10:29.314738 1516 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260808 01:10:29.323083 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 83649.6us
I20260808 01:10:29.322772 1524 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:29.326087 1524 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:29.330286 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:29.331018 1599 raft_consensus.cc:3032] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:29.331279 1599 raft_consensus.cc:729] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:29.331993 1599 consensus_queue.cc:260] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:29.333222 1599 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:29.344587 3330 raft_consensus.cc:2932] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:29.350018 1599 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:29.337617 3184 raft_consensus.cc:979] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:29.354296 1489 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:29.347628 3337 tablet_replica.cc:330] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5: stopping tablet replica
I20260808 01:10:29.360934 1598 raft_consensus.cc:2932] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:29.369330 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:29.369977 2206 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:29.360661 3334 raft_consensus.cc:1067] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:29.325831 3325 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.027s
I20260808 01:10:29.376765 2206 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:29.388531 3311 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:29.379722 3324 raft_consensus.cc:2781] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:29.412487 3324 raft_consensus.cc:686] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:29.413489 3324 consensus_queue.cc:237] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:29.434068 3337 raft_consensus.cc:2229] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:29.435164 3337 raft_consensus.cc:2258] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:29.441208 3337 ts_tablet_manager.cc:1905] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:29.446265 1439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6bb0551ce74437284392981d2921446"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55042
I20260808 01:10:29.446904 1439 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:29.447258 1439 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:29.499037 1439 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:29.493971 3309 ts_tablet_manager.cc:1918] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:29.502460 1439 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:29.504186 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:29.504410 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:29.504861 1524 raft_consensus.cc:3032] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:29.504881 1598 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:29.505319 1524 raft_consensus.cc:729] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:29.506055 1524 consensus_queue.cc:260] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:29.508030 1524 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:29.486958 2201 raft_consensus.cc:2932] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:29.514365 1598 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:29.515849 1524 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:29.517596 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2355fe5dfc824ec78c044695be9d32de with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:29.519600 1410 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:29.520509 3267 raft_consensus.cc:2781] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:29.520998 3267 raft_consensus.cc:686] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:29.521765 3267 consensus_queue.cc:237] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:29.529155 3309 log.cc:1196] T 4f0008d696f943a3aaa520463af63298 P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root/wals/4f0008d696f943a3aaa520463af63298
I20260808 01:10:29.536856 1329 catalog_manager.cc:4841] TS 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): tablet 4f0008d696f943a3aaa520463af63298 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:29.541064 3339 raft_consensus.cc:2932] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:29.543819 1600 raft_consensus.cc:2932] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:29.551272 1342 catalog_manager.cc:5495] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 3 to 4, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 4 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:29.579807 1342 catalog_manager.cc:5495] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:29.588862 1340 catalog_manager.cc:5495] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index 4 to 5, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New cstate: current_term: 2 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:29.605180 1525 raft_consensus.cc:2932] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:29.607919 1429 tablet_service.cc:1514] Processing DeleteTablet for tablet 2355fe5dfc824ec78c044695be9d32de with delete_type TABLET_DATA_TOMBSTONED (TS 3fb1a6380d27401a831cfe5f86a4cc9d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55036
I20260808 01:10:29.608958 3309 tablet_replica.cc:330] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d: stopping tablet replica
I20260808 01:10:29.609959 3309 raft_consensus.cc:2229] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:29.610785 3309 raft_consensus.cc:2258] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:29.613881 3309 ts_tablet_manager.cc:1905] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:29.641386 1444 raft_consensus.cc:2932] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:29.649173 1340 catalog_manager.cc:5495] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 2 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65), VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66), VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67), VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 3 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:29.670886 3337 ts_tablet_manager.cc:1918] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260808 01:10:29.736431 3337 log.cc:1196] T a668c2efd2dd4f5b946b70904c143c17 P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-1-root/wals/a668c2efd2dd4f5b946b70904c143c17
I20260808 01:10:29.750022 1328 catalog_manager.cc:4841] TS 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): tablet a668c2efd2dd4f5b946b70904c143c17 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:29.773985 2201 consensus_queue.cc:237] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:29.776932 3309 ts_tablet_manager.cc:1918] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260808 01:10:29.777486 3309 log.cc:1196] T 2355fe5dfc824ec78c044695be9d32de P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root/wals/2355fe5dfc824ec78c044695be9d32de
I20260808 01:10:29.779151 1329 catalog_manager.cc:4841] TS 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): tablet 2355fe5dfc824ec78c044695be9d32de (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:29.781642 1440 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:29.781642 1600 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:29.783344 3242 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:10:29.785346 3242 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:10:29.792353 3311 raft_consensus.cc:2932] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 4 to 6, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:29.801318 1440 raft_consensus.cc:2932] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 4 to 6, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:29.816824 1604 raft_consensus.cc:2932] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 4 to 6, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:29.820094 1326 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 516db2422f0644aebfd1999735efdbb8 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:29.827602 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:29.833523 1340 catalog_manager.cc:5495] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 4 to 6, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New cstate: current_term: 3 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:29.846822 1511 tablet_service.cc:1514] Processing DeleteTablet for tablet 516db2422f0644aebfd1999735efdbb8 with delete_type TABLET_DATA_TOMBSTONED (TS 6b09dbefd24b4d52b385c4dff5697dc5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58250
I20260808 01:10:29.878238 3337 tablet_replica.cc:330] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5: stopping tablet replica
I20260808 01:10:29.879242 3337 raft_consensus.cc:2229] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:29.886893 3337 raft_consensus.cc:2258] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:29.895582 3337 ts_tablet_manager.cc:1905] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:29.924227 1440 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:29.924939 1440 raft_consensus.cc:595] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:29.928772 1440 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:29.929324 1440 raft_consensus.cc:595] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:29.930799 3337 ts_tablet_manager.cc:1918] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260808 01:10:29.932013 3337 log.cc:1196] T 516db2422f0644aebfd1999735efdbb8 P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-1-root/wals/516db2422f0644aebfd1999735efdbb8
I20260808 01:10:29.932735 1525 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:29.933367 1525 raft_consensus.cc:595] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:29.934437 1328 catalog_manager.cc:4841] TS 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): tablet 516db2422f0644aebfd1999735efdbb8 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:29.937136 1440 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f6bb0551ce74437284392981d2921446"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:29.937758 1440 raft_consensus.cc:595] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:29.948042 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260808 01:10:29.949828 2206 raft_consensus.cc:1261] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:10:29.949846 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:29.951375 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:29.960052 3172 consensus_queue.cc:1035] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:10:29.974555 1604 raft_consensus.cc:1261] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:29.985157 3316 consensus_queue.cc:1035] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:29.997004 1600 raft_consensus.cc:1261] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:10:29.999133 3172 consensus_queue.cc:1035] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:10:30.011312 3267 raft_consensus.cc:979] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:30.012053 3179 raft_consensus.cc:1067] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Signalling peer a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:30.014623 1604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6bb0551ce74437284392981d2921446"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:45628
I20260808 01:10:30.015856 1604 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:30.017318 1604 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:30.032155 1525 raft_consensus.cc:1261] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:30.043704 1604 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:30.059592 1598 raft_consensus.cc:1226] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:30.057201 3316 consensus_queue.cc:1035] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260808 01:10:30.101614 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:30.102753 1604 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:30.103940 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:30.104671 1516 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:30.109616 1440 raft_consensus.cc:3032] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:30.110190 1440 raft_consensus.cc:729] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:30.111342 3329 consensus_queue.cc:1046] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:30.112239 3172 raft_consensus.cc:979] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:30.112726 3172 raft_consensus.cc:1067] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Signalling peer a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:30.115605 1604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:45638
I20260808 01:10:30.116251 1604 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:30.116629 1604 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:30.125056 1516 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:30.126606 1567 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:30.127604 3216 raft_consensus.cc:2781] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:30.128274 3216 raft_consensus.cc:686] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:30.129209 3216 consensus_queue.cc:237] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:30.121623 1440 consensus_queue.cc:260] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:30.130712 1440 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:30.145543 1604 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:30.150775 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:30.151444 1525 raft_consensus.cc:3032] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:30.151732 1525 raft_consensus.cc:729] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:30.152513 1525 consensus_queue.cc:260] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:30.153942 1525 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:30.157596 1340 catalog_manager.cc:5495] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 2 to 3, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 3 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:10:30.159930 1604 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:30.164595 1440 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
W20260808 01:10:30.165271 3356 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260808 01:10:30.167479 1524 raft_consensus.cc:1226] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 01:10:30.229211 1525 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:30.238166 1567 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:30.238722 3356 raft_consensus.cc:979] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:30.239107 3237 raft_consensus.cc:2781] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:30.239370 3355 raft_consensus.cc:1067] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Signalling peer a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:30.239641 3237 raft_consensus.cc:686] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:30.241503 1604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:45628
I20260808 01:10:30.242101 1604 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:30.242460 1604 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:30.245662 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:30.246286 2206 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:30.240651 3237 consensus_queue.cc:237] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:30.252959 1604 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:30.254635 2206 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:30.261076 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:30.261757 1440 raft_consensus.cc:3032] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:30.262081 1440 raft_consensus.cc:729] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:30.262957 1604 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:30.262750 1440 consensus_queue.cc:260] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:30.264020 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:30.265003 1516 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:30.264072 1440 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:30.275558 1516 raft_consensus.cc:2454] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:10:30.276743 1567 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:30.277601 3237 raft_consensus.cc:2781] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:30.276875 1440 raft_consensus.cc:2454] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:10:30.278482 1340 catalog_manager.cc:5495] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 2 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:30.279783 3237 raft_consensus.cc:686] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:30.289333 3237 consensus_queue.cc:237] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:30.426769 1340 catalog_manager.cc:5495] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 3 to 4, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 4 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:10:30.460417 3356 raft_consensus.cc:979] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:30.460947 3316 raft_consensus.cc:1067] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:30.466040 1525 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58266
I20260808 01:10:30.466662 1525 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:30.467067 1525 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:30.475420 1525 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:30.498819 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:30.499624 1440 raft_consensus.cc:3032] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:30.501837 1440 raft_consensus.cc:729] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:30.502243 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:30.503095 2206 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:30.511241 2206 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:30.503688 1440 consensus_queue.cc:260] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:30.513028 1440 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:30.503034 1525 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:30.518456 1487 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:30.519654 3260 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:30.520387 3260 raft_consensus.cc:686] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:30.521343 3260 consensus_queue.cc:237] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:30.527818 1440 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:30.671530 2206 raft_consensus.cc:1261] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:30.673135 1524 raft_consensus.cc:1261] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:30.674645 3216 consensus_queue.cc:1035] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:30.681227 3133 consensus_queue.cc:1035] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:30.725128 1340 catalog_manager.cc:5495] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 2 to 3, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:30.742503 1439 raft_consensus.cc:1261] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:30.743040 1525 raft_consensus.cc:1261] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:10:30.750346 3133 consensus_queue.cc:1035] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260808 01:10:30.765561 3363 consensus_queue.cc:1035] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:30.777365 1522 raft_consensus.cc:1261] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:30.782470 3270 consensus_queue.cc:1035] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:30.850898 1439 raft_consensus.cc:1261] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260808 01:10:30.914646 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 64906.8us
I20260808 01:10:30.948523 3373 consensus_queue.cc:1035] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:31.139448 2206 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:31.148778 3260 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:10:31.195394 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:31.260427 1440 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:31.269699 1444 consensus_queue.cc:237] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:31.297621 3378 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260808 01:10:31.311674 1408 consensus_peers.cc:487] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f18b5317cac4512ad16a64664fab357. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:31.386709 1516 raft_consensus.cc:1261] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:31.389498 3380 consensus_queue.cc:1035] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:31.446135 3380 raft_consensus.cc:2932] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:31.447804 1524 raft_consensus.cc:2932] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:31.455363 1519 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:31.459139 1519 raft_consensus.cc:595] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:31.477061 1600 raft_consensus.cc:1261] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:31.483233 1600 consensus_queue.cc:237] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:31.487789 1601 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4f0008d696f943a3aaa520463af63298"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:31.488587 1601 raft_consensus.cc:595] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:31.489378 3380 consensus_queue.cc:1035] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:31.494788 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 01:10:31.496666 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:31.497421 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:31.543634 1604 raft_consensus.cc:2932] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
W20260808 01:10:31.583176 1565 consensus_peers.cc:487] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a27b15da83cb42d49d1b0a703b5955ea. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:31.587586 1444 raft_consensus.cc:1261] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:31.588603 1524 raft_consensus.cc:1261] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:31.590207 3373 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:31.593329 3372 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:31.605489 1342 catalog_manager.cc:5495] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:31.619059 3372 raft_consensus.cc:979] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:31.619906 3374 raft_consensus.cc:1067] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:31.627512 1522 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f0008d696f943a3aaa520463af63298"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58280
I20260808 01:10:31.628201 1522 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:31.628572 1522 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:31.630369 3374 raft_consensus.cc:2932] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:31.636642 1522 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:31.638597 1522 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:31.639680 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:31.640409 2204 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:31.649588 1524 raft_consensus.cc:2932] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:31.652925 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:31.653642 1600 raft_consensus.cc:3032] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:31.653988 1600 raft_consensus.cc:729] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:31.654767 1600 consensus_queue.cc:260] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:31.656620 1600 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:31.669915 1600 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:31.672401 1444 raft_consensus.cc:2932] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:31.681814 1489 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:31.682964 3172 raft_consensus.cc:2781] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:31.683516 3172 raft_consensus.cc:686] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:31.684506 3172 consensus_queue.cc:237] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:31.693568 2204 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:31.750579 1440 consensus_queue.cc:237] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:31.758661 1340 catalog_manager.cc:5495] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 2 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:31.766171 1340 catalog_manager.cc:5495] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 1 to 2, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 2 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:31.779639 1600 raft_consensus.cc:1261] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:31.784986 3380 consensus_queue.cc:1035] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:31.808109 1522 raft_consensus.cc:1261] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:31.809525 3380 consensus_queue.cc:1035] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:31.828503 3362 raft_consensus.cc:2932] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:31.833135 3184 raft_consensus.cc:979] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:31.834137 3184 raft_consensus.cc:1067] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:31.835969 1445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55042
I20260808 01:10:31.836707 1445 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:31.837145 1445 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:10:31.837265 1408 consensus_peers.cc:487] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 821d5636d01e4067bac2bbd7475921ef. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:31.847492 1600 raft_consensus.cc:2932] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:31.854635 1524 raft_consensus.cc:2932] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:31.858670 1445 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:31.860618 1445 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:31.862782 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:31.863484 2204 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:31.865998 1604 consensus_queue.cc:237] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:31.868765 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:31.874143 1522 raft_consensus.cc:3032] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:31.874500 1522 raft_consensus.cc:729] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:31.875279 1522 consensus_queue.cc:260] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:31.876701 1522 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:31.868625 1340 catalog_manager.cc:5495] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:31.894345 1522 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:31.901050 3396 ts_tablet_manager.cc:927] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
W20260808 01:10:31.920454 1565 consensus_peers.cc:487] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36f2393bca80434dbdc089722b52edd1. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:31.921861 3396 tablet_copy_client.cc:316] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.65:38291
I20260808 01:10:31.924172 1458 tablet_copy_service.cc:140] P 3fb1a6380d27401a831cfe5f86a4cc9d: Received BeginTabletCopySession request for tablet 3f18b5317cac4512ad16a64664fab357 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:60392)
I20260808 01:10:31.924837 1458 tablet_copy_service.cc:161] P 3fb1a6380d27401a831cfe5f86a4cc9d: Beginning new tablet copy session on tablet 3f18b5317cac4512ad16a64664fab357 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:60392: session id = 95b661b03f1d460b9657e61d4b0247cd-3f18b5317cac4512ad16a64664fab357
I20260808 01:10:31.926931 1522 raft_consensus.cc:1261] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:31.928556 3374 consensus_queue.cc:1035] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:31.934909 1445 raft_consensus.cc:1261] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:10:31.936434 3374 consensus_queue.cc:1035] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 01:10:31.946646 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 52873us
I20260808 01:10:31.948269 1458 tablet_copy_source_session.cc:205] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:31.952488 3396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f18b5317cac4512ad16a64664fab357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:31.973477 1410 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:31.992199 3379 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:31.992933 3379 raft_consensus.cc:686] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:31.994172 3379 consensus_queue.cc:237] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:32.022089 3401 ts_tablet_manager.cc:927] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:10:32.027329 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 52505.6us
I20260808 01:10:32.029119 3396 tablet_copy_client.cc:791] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:32.032263 3401 tablet_copy_client.cc:316] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.67:35675
I20260808 01:10:32.033258 3396 tablet_copy_client.cc:655] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:32.037150 1613 tablet_copy_service.cc:140] P a4862d0938c84fc4bd840e4c17acae78: Received BeginTabletCopySession request for tablet a27b15da83cb42d49d1b0a703b5955ea from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:34182)
I20260808 01:10:32.037741 1613 tablet_copy_service.cc:161] P a4862d0938c84fc4bd840e4c17acae78: Beginning new tablet copy session on tablet a27b15da83cb42d49d1b0a703b5955ea from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:34182: session id = 95b661b03f1d460b9657e61d4b0247cd-a27b15da83cb42d49d1b0a703b5955ea
I20260808 01:10:32.044488 2204 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:32.052778 1613 tablet_copy_source_session.cc:205] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:32.054746 3396 tablet_copy_client.cc:531] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:32.061412 3401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a27b15da83cb42d49d1b0a703b5955ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:32.069919 3373 raft_consensus.cc:2932] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:32.075232 1444 raft_consensus.cc:2932] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:32.097589 1524 raft_consensus.cc:2932] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:32.109782 3396 tablet_bootstrap.cc:492] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:32.121821 1342 catalog_manager.cc:5495] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index -1 to 2, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:32.146778 3401 tablet_copy_client.cc:791] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:32.149070 1339 catalog_manager.cc:5495] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 3 to 4, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 4 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:32.166604 3401 tablet_copy_client.cc:655] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:32.177218 3401 tablet_copy_client.cc:531] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:32.196795 1600 raft_consensus.cc:1261] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:32.199051 3401 tablet_bootstrap.cc:492] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:32.203814 3396 tablet_bootstrap.cc:492] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:32.208251 3172 consensus_queue.cc:1035] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:32.208757 3396 tablet_bootstrap.cc:492] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:32.209761 3396 ts_tablet_manager.cc:1397] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.100s user 0.023s sys 0.004s
I20260808 01:10:32.213634 3396 raft_consensus.cc:348] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:32.214931 3396 raft_consensus.cc:729] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:32.216878 3396 consensus_queue.cc:260] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:32.219125 3396 ts_tablet_manager.cc:1428] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260808 01:10:32.230448 1458 tablet_copy_service.cc:342] P 3fb1a6380d27401a831cfe5f86a4cc9d: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-3f18b5317cac4512ad16a64664fab357 received from {username='slave'} at 127.0.0.1:60392
I20260808 01:10:32.230890 1458 tablet_copy_service.cc:433] P 3fb1a6380d27401a831cfe5f86a4cc9d: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-3f18b5317cac4512ad16a64664fab357 on tablet 3f18b5317cac4512ad16a64664fab357 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:32.294727 3401 tablet_bootstrap.cc:492] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:32.299945 3401 tablet_bootstrap.cc:492] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:32.302250 3401 ts_tablet_manager.cc:1397] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.103s user 0.022s sys 0.020s
I20260808 01:10:32.298053 2204 raft_consensus.cc:1261] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:32.315928 3401 raft_consensus.cc:348] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:32.317273 3401 raft_consensus.cc:729] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:32.318095 3401 consensus_queue.cc:260] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:32.325208 3172 consensus_queue.cc:1035] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:32.325758 3401 ts_tablet_manager.cc:1428] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260808 01:10:32.326534 3396 ts_tablet_manager.cc:927] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:32.334470 3396 tablet_copy_client.cc:316] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.67:35675
W20260808 01:10:32.354761 1470 debug-util.cc:398] Leaking SignalData structure 0x7b080011a540 after lost signal to thread 32596
W20260808 01:10:32.356720 1470 debug-util.cc:398] Leaking SignalData structure 0x7b08001fb880 after lost signal to thread 1376
W20260808 01:10:32.358078 1470 debug-util.cc:398] Leaking SignalData structure 0x7b080034e420 after lost signal to thread 1473
W20260808 01:10:32.360569 1470 debug-util.cc:398] Leaking SignalData structure 0x7b080023f320 after lost signal to thread 1551
W20260808 01:10:32.361954 1470 debug-util.cc:398] Leaking SignalData structure 0x7b0800336960 after lost signal to thread 1629
W20260808 01:10:32.363327 1470 debug-util.cc:398] Leaking SignalData structure 0x7b0800360920 after lost signal to thread 2233
I20260808 01:10:32.384706 1613 tablet_copy_service.cc:342] P a4862d0938c84fc4bd840e4c17acae78: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-a27b15da83cb42d49d1b0a703b5955ea received from {username='slave'} at 127.0.0.1:34182
I20260808 01:10:32.385253 1613 tablet_copy_service.cc:433] P a4862d0938c84fc4bd840e4c17acae78: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-a27b15da83cb42d49d1b0a703b5955ea on tablet a27b15da83cb42d49d1b0a703b5955ea with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:32.392858 2209 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:10:32.396760 3379 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:10:32.397231 1613 tablet_copy_service.cc:140] P a4862d0938c84fc4bd840e4c17acae78: Received BeginTabletCopySession request for tablet 36f2393bca80434dbdc089722b52edd1 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:34182)
I20260808 01:10:32.397806 1613 tablet_copy_service.cc:161] P a4862d0938c84fc4bd840e4c17acae78: Beginning new tablet copy session on tablet 36f2393bca80434dbdc089722b52edd1 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:34182: session id = 95b661b03f1d460b9657e61d4b0247cd-36f2393bca80434dbdc089722b52edd1
I20260808 01:10:32.410770 3401 ts_tablet_manager.cc:927] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291)
I20260808 01:10:32.448993 2201 raft_consensus.cc:1203] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:32.461745 1613 tablet_copy_source_session.cc:205] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:32.480556 3401 tablet_copy_client.cc:316] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.65:38291
I20260808 01:10:32.484861 1458 tablet_copy_service.cc:140] P 3fb1a6380d27401a831cfe5f86a4cc9d: Received BeginTabletCopySession request for tablet 821d5636d01e4067bac2bbd7475921ef from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:60392)
I20260808 01:10:32.501784 1525 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:10:32.504082 3379 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:10:32.504491 1458 tablet_copy_service.cc:161] P 3fb1a6380d27401a831cfe5f86a4cc9d: Beginning new tablet copy session on tablet 821d5636d01e4067bac2bbd7475921ef from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:60392: session id = 95b661b03f1d460b9657e61d4b0247cd-821d5636d01e4067bac2bbd7475921ef
I20260808 01:10:32.561853 1458 tablet_copy_source_session.cc:205] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:32.567519 3401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 821d5636d01e4067bac2bbd7475921ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:32.611039 3401 tablet_copy_client.cc:791] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:32.611662 3401 tablet_copy_client.cc:655] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:32.618505 3401 tablet_copy_client.cc:531] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260808 01:10:32.619283 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 171035us
I20260808 01:10:32.635867 3401 tablet_bootstrap.cc:492] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:32.636529 2206 raft_consensus.cc:1203] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260808 01:10:32.645422 3396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36f2393bca80434dbdc089722b52edd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:32.652602 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:32.777773 3396 tablet_copy_client.cc:791] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:32.779621 3396 tablet_copy_client.cc:655] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:32.799489 3396 tablet_copy_client.cc:531] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:32.809660 3396 tablet_bootstrap.cc:492] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:32.812223 1440 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:32.812983 1440 raft_consensus.cc:595] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:32.817062 1525 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:32.817732 1525 raft_consensus.cc:595] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:32.819063 3401 tablet_bootstrap.cc:492] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:32.820039 3401 tablet_bootstrap.cc:492] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:32.820824 3401 ts_tablet_manager.cc:1397] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.185s user 0.026s sys 0.003s
I20260808 01:10:32.823360 3401 raft_consensus.cc:348] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:32.841645 3401 raft_consensus.cc:729] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:32.842424 3401 consensus_queue.cc:260] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:32.844458 1598 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:32.845196 1598 raft_consensus.cc:595] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:32.860481 3401 ts_tablet_manager.cc:1428] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.039s user 0.002s sys 0.007s
I20260808 01:10:32.863960 1458 tablet_copy_service.cc:342] P 3fb1a6380d27401a831cfe5f86a4cc9d: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-821d5636d01e4067bac2bbd7475921ef received from {username='slave'} at 127.0.0.1:60392
I20260808 01:10:32.864427 1458 tablet_copy_service.cc:433] P 3fb1a6380d27401a831cfe5f86a4cc9d: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-821d5636d01e4067bac2bbd7475921ef on tablet 821d5636d01e4067bac2bbd7475921ef with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:32.907747 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260808 01:10:32.924715 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:32.925868 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:32.995411 2206 raft_consensus.cc:1203] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:33.025870 3362 raft_consensus.cc:1050] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:33.035429 3374 raft_consensus.cc:1050] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:33.038349 3374 consensus_queue.cc:237] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:33.046414 3362 consensus_queue.cc:237] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:33.066080 1524 raft_consensus.cc:1261] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:33.068332 1445 raft_consensus.cc:1261] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:33.070225 3372 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:33.071024 2206 raft_consensus.cc:1261] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:10:33.072659 3372 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:33.098086 3423 raft_consensus.cc:979] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:33.099176 3380 raft_consensus.cc:1067] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Signalling peer a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:33.101315 1598 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:45628
I20260808 01:10:33.101928 1598 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:33.102324 1598 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:33.124055 2206 raft_consensus.cc:1261] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:33.124470 3425 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260808 01:10:33.128715 3380 consensus_queue.cc:1035] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:33.129472 3429 raft_consensus.cc:2932] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:33.131922 1445 raft_consensus.cc:2932] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:33.174645 1598 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:33.176885 1600 raft_consensus.cc:1261] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:33.186220 1524 raft_consensus.cc:1261] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:33.191025 1522 raft_consensus.cc:2932] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:33.232138 3429 raft_consensus.cc:979] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:33.232643 3133 raft_consensus.cc:1067] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:33.242187 1339 catalog_manager.cc:5495] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:33.250545 1444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55040
I20260808 01:10:33.251168 1444 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:33.251555 1444 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:33.251132 2206 raft_consensus.cc:2932] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:33.265411 3356 consensus_queue.cc:1035] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:33.261300 1444 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:33.270633 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:33.271346 2201 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:33.279974 2201 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:33.287295 1408 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:33.288370 3362 raft_consensus.cc:2781] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:33.288872 3362 raft_consensus.cc:686] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:33.289917 3362 consensus_queue.cc:237] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:33.280792 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:33.293949 1600 raft_consensus.cc:3032] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:33.294348 1600 raft_consensus.cc:729] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:33.295326 1600 consensus_queue.cc:260] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:33.296717 1600 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:33.298329 1598 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:33.285290 1444 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:33.225203 3396 tablet_bootstrap.cc:492] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:33.310573 3396 tablet_bootstrap.cc:492] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:33.312569 3172 raft_consensus.cc:979] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:33.312783 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:33.331969 1600 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:33.345966 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:33.358517 3172 raft_consensus.cc:1067] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:33.359099 2206 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:33.358521 1440 raft_consensus.cc:3032] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:33.360807 1440 raft_consensus.cc:729] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:33.361553 1440 consensus_queue.cc:260] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:33.363255 1440 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:33.364450 3380 consensus_queue.cc:1035] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:33.375145 2206 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:33.376832 3396 ts_tablet_manager.cc:1397] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.567s user 0.018s sys 0.009s
I20260808 01:10:33.377635 1440 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:33.398509 1568 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:33.400873 3429 raft_consensus.cc:2781] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:33.401423 3429 raft_consensus.cc:686] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:33.402364 1445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55042
I20260808 01:10:33.402948 1445 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:33.403288 1445 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:33.402704 3429 consensus_queue.cc:237] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:33.396386 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 44886.3us
I20260808 01:10:33.410765 1445 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:33.415267 1445 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:33.395936 3396 raft_consensus.cc:348] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:33.511298 3380 raft_consensus.cc:1050] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
W20260808 01:10:33.513770 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 76151.8us
I20260808 01:10:33.472843 3396 raft_consensus.cc:729] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
W20260808 01:10:33.516192 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 84384.6us
I20260808 01:10:33.522136 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:33.522746 1524 raft_consensus.cc:3032] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:33.523011 1524 raft_consensus.cc:729] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:33.526821 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:33.527601 1601 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:33.540170 3396 consensus_queue.cc:260] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:33.542426 3396 ts_tablet_manager.cc:1428] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.159s user 0.000s sys 0.007s
I20260808 01:10:33.546698 3380 consensus_queue.cc:237] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:33.564654 1524 consensus_queue.cc:260] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:33.567021 1524 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:33.587363 1601 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:33.589387 3379 raft_consensus.cc:2932] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:33.588429 1338 catalog_manager.cc:5495] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 2 to 3, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 3 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:33.597756 1524 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
W20260808 01:10:33.607689 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 78300.3us
I20260808 01:10:33.610211 1410 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:33.612139 1613 tablet_copy_service.cc:342] P a4862d0938c84fc4bd840e4c17acae78: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-36f2393bca80434dbdc089722b52edd1 received from {username='slave'} at 127.0.0.1:34182
I20260808 01:10:33.612666 1613 tablet_copy_service.cc:433] P a4862d0938c84fc4bd840e4c17acae78: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-36f2393bca80434dbdc089722b52edd1 on tablet 36f2393bca80434dbdc089722b52edd1 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:33.620442 2201 raft_consensus.cc:1203] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260808 01:10:33.622779 1525 raft_consensus.cc:2932] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:33.629536 3362 raft_consensus.cc:2781] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:33.664779 1338 catalog_manager.cc:5495] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 3 to 4, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 4 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:33.715569 1340 catalog_manager.cc:5495] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:33.752030 3362 raft_consensus.cc:686] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:33.760905 3362 consensus_queue.cc:237] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:10:33.838573 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 57677.8us
W20260808 01:10:33.838639 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 206404us
I20260808 01:10:33.837828 1604 consensus_queue.cc:237] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:33.928488 2201 raft_consensus.cc:1261] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:33.971767 1440 raft_consensus.cc:1261] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:33.996178 2209 raft_consensus.cc:1261] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:33.997572 3425 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:10:34.013540 3374 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260808 01:10:34.033797 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 195818us
I20260808 01:10:34.057518 1600 raft_consensus.cc:2932] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:34.073848 1601 raft_consensus.cc:1261] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:34.092933 1598 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:10:34.095060 1338 catalog_manager.cc:5495] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 2 to 3, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 3 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
W20260808 01:10:34.149832 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 177084us
W20260808 01:10:34.150265 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 166510us
W20260808 01:10:34.159090 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 185775us
I20260808 01:10:34.141933 3467 consensus_queue.cc:1035] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260808 01:10:34.191251 1525 raft_consensus.cc:1261] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260808 01:10:34.209969 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 56221.7us
I20260808 01:10:34.153621 3457 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260808 01:10:34.239114 2207 raft_consensus.cc:1261] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:34.238034 2204 raft_consensus.cc:2932] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:34.169103 3466 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
W20260808 01:10:34.248732 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 61371us
W20260808 01:10:34.172317 3460 consensus_queue.cc:1159] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a4862d0938c84fc4bd840e4c17acae78" responder_term: 1 status { last_received { term: 1 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 1 index: 3 } }
I20260808 01:10:34.256420 3460 consensus_queue.cc:1035] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:34.272677 2209 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:10:34.275349 1440 consensus_queue.cc:237] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:34.306030 3467 consensus_queue.cc:1035] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:10:34.326130 3469 raft_consensus.cc:2932] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:34.316800 3362 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260808 01:10:34.334007 3456 raft_consensus.cc:1050] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:34.338992 1524 raft_consensus.cc:1261] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:34.337005 2209 raft_consensus.cc:2932] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:34.375121 1524 raft_consensus.cc:1261] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:34.380846 1440 raft_consensus.cc:1261] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:34.381597 3464 consensus_queue.cc:1035] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20260808 01:10:34.384410 3464 consensus_queue.cc:1035] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:34.410244 3464 raft_consensus.cc:2932] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:34.415253 1598 raft_consensus.cc:1261] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:34.417706 1525 raft_consensus.cc:2932] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:34.420599 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:34.445291 3133 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:10:34.453804 2207 raft_consensus.cc:1261] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:34.417809 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a27b15da83cb42d49d1b0a703b5955ea with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:10:34.454900 1328 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 51932us
I20260808 01:10:34.507535 3456 consensus_queue.cc:237] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:34.550726 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 163066us
I20260808 01:10:34.551234 1596 raft_consensus.cc:2932] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:34.620234 1340 catalog_manager.cc:5495] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: config changed from index 4 to 5, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New cstate: current_term: 2 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:34.575160 3380 consensus_queue.cc:1035] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.076s
I20260808 01:10:34.579972 3460 consensus_queue.cc:1035] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260808 01:10:34.764856 1442 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:34.769287 1442 raft_consensus.cc:595] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:34.799868 1525 raft_consensus.cc:2932] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:34.529695 3443 raft_consensus.cc:2932] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:10:34.814810 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 199190us
W20260808 01:10:34.832994 3475 log.cc:924] Time spent T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
W20260808 01:10:34.902726 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 45856.2us
I20260808 01:10:34.934792 1439 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:34.937937 1439 raft_consensus.cc:595] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:34.943460 2206 raft_consensus.cc:2932] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:34.948909 2209 raft_consensus.cc:2932] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:10:34.969036 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 113276us
W20260808 01:10:35.041136 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 68169.8us
W20260808 01:10:35.053360 3497 log.cc:924] Time spent T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20260808 01:10:35.063040 1444 raft_consensus.cc:1261] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:35.086676 2208 raft_consensus.cc:1261] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 1 LEARNER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:35.088153 1599 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:35.094908 1599 raft_consensus.cc:595] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
W20260808 01:10:35.165597 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 56771.6us
W20260808 01:10:35.172293 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 72461.9us
I20260808 01:10:35.096478 3469 consensus_queue.cc:1035] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.094s
I20260808 01:10:35.201546 1519 raft_consensus.cc:1261] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:10:35.219079 1442 raft_consensus.cc:2932] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:10:35.279301 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 174790us
W20260808 01:10:35.321362 3449 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.012s sys 0.016s
W20260808 01:10:35.321887 3449 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.721s user 0.012s sys 0.016s
W20260808 01:10:35.312098 3485 log.cc:924] Time spent T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260808 01:10:35.384150 3462 consensus_queue.cc:1035] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.135s
I20260808 01:10:35.513695 3451 raft_consensus.cc:979] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:35.514400 3451 raft_consensus.cc:1067] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
W20260808 01:10:35.536696 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 427495us
I20260808 01:10:35.564716 3429 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:35.571887 3429 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:35.556672 3443 consensus_meta.cc:287] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.646s user 0.009s sys 0.004s
I20260808 01:10:35.570183 3460 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:35.576669 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:35.601593 3374 consensus_queue.cc:1035] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:35.577162 3460 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:35.588523 3429 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:35.617084 3460 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:35.629915 1524 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58266
I20260808 01:10:35.630235 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:35.630705 1524 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:35.631227 1524 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:10:35.658015 1489 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.006s sys 0.026s
W20260808 01:10:35.674598 1489 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.571s user 0.007s sys 0.030s
W20260808 01:10:35.675041 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 572790us
I20260808 01:10:35.679909 3443 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:55036 (request call id 847) took 1530 ms. Trace:
I20260808 01:10:35.680608 3443 rpcz_store.cc:276] 0808 01:10:34.144470 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:34.144634 (+ 164us) service_pool.cc:227] Handling call
0808 01:10:34.274596 (+129962us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b58002306b0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34c9baf63 00007fe34ca1eaba 00007fe34ca3aa7a 00007fe34ca3bada 00007fe34ca3ac8b 00007fe34ca46084 00007fe34ca45743 00007fe34ca3dc1a 00007fe3519816d9 00007fe34b06b184
0808 01:10:34.278116 (+ 3520us) log.cc:842] Serialized 501 byte log entry
0808 01:10:35.645657 (+1367541us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42820,"raft.queue_time_us":51317,"raft.run_cpu_time_us":957,"raft.run_wall_time_us":955,"spinlock_wait_cycles":231481728}
I20260808 01:10:35.681988 3362 raft_consensus.cc:979] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:35.682755 3362 raft_consensus.cc:1067] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Signalling peer a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:35.685914 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:35.688103 1524 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:35.685041 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 49187.2us
I20260808 01:10:35.702988 1524 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:35.723229 1329 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3f18b5317cac4512ad16a64664fab357 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:35.726089 3324 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:35.731196 3324 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:35.731985 1518 raft_consensus.cc:1226] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 4. Current term is 5. Ops: []
I20260808 01:10:35.734010 3324 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:35.738951 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:35.740267 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:35.741935 2200 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:35.745095 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:10:35.749918 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 49145.5us
I20260808 01:10:35.769846 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:35.776271 1568 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:35.788776 3512 raft_consensus.cc:2726] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:35.789821 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260808 01:10:35.797011 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:35.800892 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:35.801550 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:35.799285 1342 catalog_manager.cc:5495] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:35.812597 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:35.813264 1440 raft_consensus.cc:3032] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:10:35.813555 1440 raft_consensus.cc:729] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:35.814247 1440 consensus_queue.cc:260] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:35.815339 1440 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:35.811585 1489 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:35.828249 3516 raft_consensus.cc:2726] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:35.843286 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:35.855433 2200 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:10:35.857311 1487 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:35.859097 3490 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:10:35.860457 3490 raft_consensus.cc:686] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
W20260808 01:10:35.861169 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 46004.6us
I20260808 01:10:35.866407 1440 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:10:35.873438 3490 consensus_queue.cc:237] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:35.889729 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 113394us
I20260808 01:10:35.893620 1340 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37234 (request call id 209) took 1294 ms. Trace:
I20260808 01:10:35.901373 1338 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55758 (request call id 298) took 1259 ms. Trace:
I20260808 01:10:35.901815 1338 rpcz_store.cc:276] 0808 01:10:34.642497 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:34.642666 (+ 169us) service_pool.cc:227] Handling call
0808 01:10:35.901312 (+1258646us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67071,"reported_tablets":1}
I20260808 01:10:35.906033 1410 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:35.908279 1511 tablet_service.cc:1514] Processing DeleteTablet for tablet a27b15da83cb42d49d1b0a703b5955ea with delete_type TABLET_DATA_TOMBSTONED (TS 6b09dbefd24b4d52b385c4dff5697dc5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58250
I20260808 01:10:35.894062 1340 rpcz_store.cc:276] 0808 01:10:34.599292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:34.599526 (+ 234us) service_pool.cc:227] Handling call
0808 01:10:35.893438 (+1293912us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 01:10:34.964682 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:10:34.965258 (+ 576us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:10:34.965304 (+ 46us) write_op.cc:435] Acquired schema lock
0808 01:10:34.965341 (+ 37us) tablet.cc:660] Decoding operations
0808 01:10:34.965796 (+ 455us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:10:34.965862 (+ 66us) write_op.cc:641] Partition lock acquired for write op
0808 01:10:34.965888 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
0808 01:10:34.966193 (+ 305us) tablet.cc:699] Row locks acquired
0808 01:10:34.966229 (+ 36us) write_op.cc:260] PREPARE: finished
0808 01:10:34.966413 (+ 184us) write_op.cc:270] Start()
0808 01:10:34.966544 (+ 131us) write_op.cc:276] Timestamp: P: 1786151434966347 usec, L: 0
0808 01:10:34.966572 (+ 28us) op_driver.cc:331] REPLICATION: starting
0808 01:10:34.967354 (+ 782us) log.cc:842] Serialized 536 byte log entry
0808 01:10:35.610477 (+643123us) op_driver.cc:444] REPLICATION: finished
0808 01:10:35.757232 (+146755us) write_op.cc:301] APPLY: starting
0808 01:10:35.757341 (+ 109us) tablet.cc:1365] starting BulkCheckPresence
0808 01:10:35.773248 (+ 15907us) tablet.cc:1368] finished BulkCheckPresence
0808 01:10:35.773395 (+ 147us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 01:10:35.773610 (+ 215us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 01:10:35.773767 (+ 157us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 01:10:35.773815 (+ 48us) write_op.cc:312] APPLY: finished
0808 01:10:35.774010 (+ 195us) log.cc:842] Serialized 23 byte log entry
0808 01:10:35.777612 (+ 3602us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:10:35.784797 (+ 7185us) write_op.cc:454] Released schema lock
0808 01:10:35.786127 (+ 1330us) write_op.cc:341] FINISH: Updating metrics
0808 01:10:35.820668 (+ 34541us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:10:35.820723 (+ 55us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2059,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":146388,"apply.run_cpu_time_us":3668,"apply.run_wall_time_us":63804,"num_ops":1,"prepare.queue_time_us":159196,"prepare.run_cpu_time_us":3058,"prepare.run_wall_time_us":3058,"replication_time_us":643726,"thread_start_us":177733,"threads_started":2}]]}
I20260808 01:10:35.916541 1598 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:45628
I20260808 01:10:35.917413 1598 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:35.917900 1598 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:35.925187 3380 raft_consensus.cc:2726] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 01:10:35.935878 3525 log.cc:924] Time spent T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.109s user 0.000s sys 0.004s
I20260808 01:10:35.980965 1598 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:35.987336 1598 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:35.996277 3458 raft_consensus.cc:2932] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:36.138878 3517 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:36.313002 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:36.313763 1440 raft_consensus.cc:3032] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:10:36.314079 1440 raft_consensus.cc:729] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:36.314810 1440 consensus_queue.cc:260] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:36.316051 1440 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:36.195541 3517 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:36.330113 3517 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:36.331809 3517 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:36.332527 3517 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:36.334487 3517 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:36.356981 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:36.360714 1525 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:36.364089 1440 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:10:36.373152 2200 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260808 01:10:36.363806 1518 raft_consensus.cc:2932] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:36.380429 1525 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:10:36.370965 2204 raft_consensus.cc:2932] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:36.402041 3172 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:10:36.420292 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:10:36.441593 3528 log.cc:924] Time spent T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.067s user 0.004s sys 0.000s
I20260808 01:10:36.448958 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:36.449148 3530 tablet_replica.cc:330] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5: stopping tablet replica
I20260808 01:10:36.451037 3530 raft_consensus.cc:2229] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:36.451972 3530 raft_consensus.cc:2258] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:36.456099 3530 ts_tablet_manager.cc:1905] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:36.471411 1567 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:36.473124 3429 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:10:36.474522 3429 raft_consensus.cc:686] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
W20260808 01:10:36.508188 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 138358us
I20260808 01:10:36.532764 3429 consensus_queue.cc:237] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:36.535354 1439 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260808 01:10:36.538323 3463 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:36.538908 3463 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:36.541193 3463 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:36.549091 1440 raft_consensus.cc:2932] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:10:36.560422 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 157415us
I20260808 01:10:36.580597 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:36.586845 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:36.593262 3172 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 01:10:36.594236 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:36.600569 1568 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:36.597153 1339 catalog_manager.cc:5495] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:36.609599 3505 raft_consensus.cc:979] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:36.610333 3470 raft_consensus.cc:1067] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:36.617413 1440 consensus_queue.cc:237] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:36.606178 3469 raft_consensus.cc:2726] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:36.628949 3531 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:36.652444 3531 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:36.656036 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:36.674273 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:36.676542 1586 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f18b5317cac4512ad16a64664fab357 with delete_type TABLET_DATA_TOMBSTONED (TS a4862d0938c84fc4bd840e4c17acae78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45612
I20260808 01:10:36.657095 3531 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:36.716096 1339 catalog_manager.cc:5495] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 4 to 5, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 5 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:36.756995 1339 catalog_manager.cc:5495] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 2 to 3, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:10:36.763501 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 125876us
I20260808 01:10:36.764336 1568 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:36.763895 1410 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:36.767269 1444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55040
I20260808 01:10:36.767856 1444 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:36.768206 1444 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:36.793113 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:36.768802 3505 raft_consensus.cc:2726] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:36.798437 1519 raft_consensus.cc:1261] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:36.800271 3454 consensus_queue.cc:1035] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:36.771028 3536 raft_consensus.cc:2726] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:36.812223 1489 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:36.823113 3516 raft_consensus.cc:2726] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:36.823179 3530 ts_tablet_manager.cc:1918] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260808 01:10:36.824564 3530 log.cc:1196] T a27b15da83cb42d49d1b0a703b5955ea P 6b09dbefd24b4d52b385c4dff5697dc5: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-1-root/wals/a27b15da83cb42d49d1b0a703b5955ea
I20260808 01:10:36.837287 1444 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:36.840293 1444 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:36.840804 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:36.841436 1522 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:36.827685 2202 raft_consensus.cc:1261] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:36.843508 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:36.844370 1604 raft_consensus.cc:3032] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:36.844676 1604 raft_consensus.cc:729] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:36.847498 1604 consensus_queue.cc:260] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:36.851423 1604 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:36.855643 3443 consensus_queue.cc:1035] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:36.874753 1604 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:36.828871 1328 catalog_manager.cc:4841] TS 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): tablet a27b15da83cb42d49d1b0a703b5955ea (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
W20260808 01:10:36.875711 1328 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 46860.3us
I20260808 01:10:36.879983 1410 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:36.883723 1522 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:36.892427 1340 catalog_manager.cc:5495] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 5 to 6, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 6 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:10:36.900766 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:36.896432 3545 tablet_replica.cc:330] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78: stopping tablet replica
I20260808 01:10:36.887387 3443 raft_consensus.cc:2781] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:36.949486 3443 raft_consensus.cc:686] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:36.951557 3443 consensus_queue.cc:237] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:36.956595 3531 raft_consensus.cc:2932] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:36.980844 2203 raft_consensus.cc:2932] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:36.968199 1519 raft_consensus.cc:2932] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:37.005851 1329 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 821d5636d01e4067bac2bbd7475921ef with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:37.022531 1598 consensus_queue.cc:237] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:37.024230 1525 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:37.026101 1525 raft_consensus.cc:595] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Received request to transfer leadership to TS 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:37.033474 1447 raft_consensus.cc:1261] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:10:37.042585 3133 consensus_queue.cc:1035] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:37.044696 2204 raft_consensus.cc:1261] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:37.046842 1604 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:37.047462 1604 raft_consensus.cc:595] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:37.051254 1525 raft_consensus.cc:1261] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:10:37.052791 3545 raft_consensus.cc:2229] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:37.084688 3545 raft_consensus.cc:2258] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:37.078130 3429 consensus_queue.cc:1035] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260808 01:10:37.088699 3133 consensus_queue.cc:1035] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:37.091729 1604 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:37.095299 1522 raft_consensus.cc:1261] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:10:37.105398 3544 raft_consensus.cc:979] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:37.108038 3545 ts_tablet_manager.cc:1905] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:37.109683 3544 raft_consensus.cc:1067] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:37.103617 1604 raft_consensus.cc:595] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:37.111603 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260808 01:10:37.111676 1440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55040
I20260808 01:10:37.112244 1440 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:37.112571 1440 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:37.113458 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:37.114161 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:37.121148 1339 catalog_manager.cc:5495] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:37.135321 1338 catalog_manager.cc:5495] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 3 to 4, VOTER a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) evicted. New cstate: current_term: 1 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:37.139561 3458 consensus_queue.cc:1035] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:37.140820 3505 raft_consensus.cc:2932] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:37.151186 1522 raft_consensus.cc:2932] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:37.158113 1440 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:37.161280 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:37.161938 1596 raft_consensus.cc:3032] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:37.162227 1596 raft_consensus.cc:729] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:37.162907 1596 consensus_queue.cc:260] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:37.164870 1596 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:37.167112 1440 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:37.168170 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:37.168783 2206 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:37.177975 3542 raft_consensus.cc:979] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:37.178524 3542 raft_consensus.cc:1067] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Signalling peer 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:37.186919 2206 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:37.191154 1596 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:37.192636 1410 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:37.193271 2202 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:49268
I20260808 01:10:37.210825 3443 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:37.211233 2202 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:37.211454 3443 raft_consensus.cc:686] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:37.212414 3443 consensus_queue.cc:237] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:37.211678 2202 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:37.226832 3469 raft_consensus.cc:979] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:37.226086 2200 raft_consensus.cc:2932] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:37.227626 3469 raft_consensus.cc:1067] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:37.257386 1440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55040
I20260808 01:10:37.257967 1440 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:37.258369 1440 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:10:37.269840 1586 tablet_service.cc:1514] Processing DeleteTablet for tablet 821d5636d01e4067bac2bbd7475921ef with delete_type TABLET_DATA_TOMBSTONED (TS a4862d0938c84fc4bd840e4c17acae78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45612
I20260808 01:10:37.289831 1440 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:37.296015 1440 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:37.297887 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:37.299300 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:37.302580 1340 catalog_manager.cc:5495] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 3 to 4, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New cstate: current_term: 1 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:37.306437 1596 raft_consensus.cc:3032] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:10:37.306843 1596 raft_consensus.cc:729] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:37.307999 1596 consensus_queue.cc:260] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:37.319218 1596 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:10:37.327068 1338 catalog_manager.cc:5495] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 2 to 3, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 3 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:37.335743 1429 tablet_service.cc:1514] Processing DeleteTablet for tablet 36f2393bca80434dbdc089722b52edd1 with delete_type TABLET_DATA_TOMBSTONED (TS 3fb1a6380d27401a831cfe5f86a4cc9d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55036
I20260808 01:10:37.347225 2202 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:37.355561 1596 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:10:37.366370 1598 raft_consensus.cc:1261] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:37.371171 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 36f2393bca80434dbdc089722b52edd1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:37.371587 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:37.372241 1519 raft_consensus.cc:3032] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:10:37.372507 1519 raft_consensus.cc:729] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:37.373135 1519 consensus_queue.cc:260] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:37.374131 1519 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:37.382058 3454 consensus_queue.cc:1035] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:37.389276 2202 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:37.390630 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:37.391294 1599 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:37.400983 1525 raft_consensus.cc:1261] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:37.401362 1519 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:10:37.402541 2171 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:37.404237 3554 raft_consensus.cc:2781] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:10:37.404306 3454 consensus_queue.cc:1035] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:37.404860 3554 raft_consensus.cc:686] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:37.405831 3554 consensus_queue.cc:237] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:37.406970 1410 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:37.407752 3443 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:10:37.408247 3443 raft_consensus.cc:686] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:37.415143 1599 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:10:37.409055 3443 consensus_queue.cc:237] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:37.426795 3545 ts_tablet_manager.cc:1918] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:37.427279 3545 log.cc:1196] T 3f18b5317cac4512ad16a64664fab357 P a4862d0938c84fc4bd840e4c17acae78: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-2-root/wals/3f18b5317cac4512ad16a64664fab357
I20260808 01:10:37.430164 1328 catalog_manager.cc:4841] TS a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): tablet 3f18b5317cac4512ad16a64664fab357 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:37.430744 3559 tablet_replica.cc:330] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d: stopping tablet replica
I20260808 01:10:37.438655 3559 raft_consensus.cc:2229] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:37.449105 1340 catalog_manager.cc:5495] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 6 to 7, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 7 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:37.442677 3559 raft_consensus.cc:2258] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:37.460570 3559 ts_tablet_manager.cc:1905] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:37.503116 3545 tablet_replica.cc:330] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78: stopping tablet replica
I20260808 01:10:37.504611 3545 raft_consensus.cc:2229] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:37.522883 3545 raft_consensus.cc:2258] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:37.526692 3545 ts_tablet_manager.cc:1905] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:37.532589 1340 catalog_manager.cc:5495] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 4 to 5, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 5 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:37.652508 3559 ts_tablet_manager.cc:1918] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:37.653048 3559 log.cc:1196] T 36f2393bca80434dbdc089722b52edd1 P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root/wals/36f2393bca80434dbdc089722b52edd1
I20260808 01:10:37.655092 1329 catalog_manager.cc:4841] TS 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): tablet 36f2393bca80434dbdc089722b52edd1 (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:37.651527 3545 ts_tablet_manager.cc:1918] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:10:37.656091 3545 log.cc:1196] T 821d5636d01e4067bac2bbd7475921ef P a4862d0938c84fc4bd840e4c17acae78: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-2-root/wals/821d5636d01e4067bac2bbd7475921ef
I20260808 01:10:37.657538 1328 catalog_manager.cc:4841] TS a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): tablet 821d5636d01e4067bac2bbd7475921ef (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:37.751911 1604 raft_consensus.cc:1261] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:37.753587 3467 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:10:37.783869 2200 raft_consensus.cc:1261] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:37.785924 3454 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:10:37.809679 1604 raft_consensus.cc:1261] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260808 01:10:37.814564 1525 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:10:37.814754 3454 consensus_queue.cc:1035] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:10:37.833688 1599 raft_consensus.cc:1261] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:10:37.837983 3532 consensus_queue.cc:1035] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:10:37.861177 1525 raft_consensus.cc:1261] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260808 01:10:37.874714 3547 consensus_queue.cc:1035] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:37.985296 1519 raft_consensus.cc:1261] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:10:37.994637 3532 consensus_queue.cc:1035] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:10:38.269368 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:38.349599 1447 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:38.350232 1447 raft_consensus.cc:595] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:38.358774 1447 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:38.359368 1447 raft_consensus.cc:595] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:38.361928 1447 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:38.362890 1447 raft_consensus.cc:595] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:38.366029 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260808 01:10:38.367858 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:38.368538 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:38.403680 3467 raft_consensus.cc:979] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:38.404258 3467 raft_consensus.cc:1067] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Signalling peer a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:38.421234 1596 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:45628
I20260808 01:10:38.421897 1596 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:38.423029 1596 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:38.446429 1596 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:38.449728 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:38.449858 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:38.450349 1447 raft_consensus.cc:3032] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:38.450508 1596 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:38.450721 1447 raft_consensus.cc:729] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:38.450591 2202 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:38.451526 1447 consensus_queue.cc:260] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:38.452989 1447 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:38.468362 1447 raft_consensus.cc:2454] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:38.469564 1568 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:38.470501 3469 raft_consensus.cc:2781] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:38.471025 3469 raft_consensus.cc:686] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:38.471964 3469 consensus_queue.cc:237] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:38.474601 2202 raft_consensus.cc:2454] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:38.501410 1340 catalog_manager.cc:5495] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 2 to 3, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 3 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:38.539654 1596 consensus_queue.cc:237] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:10:38.585479 1565 consensus_peers.cc:487] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1355c134ca5f45b381fb2b12b1865adb. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:38.587333 1447 raft_consensus.cc:1261] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 01:10:38.656406 1519 raft_consensus.cc:1261] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 01:10:38.663424 3454 raft_consensus.cc:979] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:38.664006 3520 raft_consensus.cc:1067] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Signalling peer a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:38.666021 1599 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:45628
I20260808 01:10:38.666551 1599 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:38.666878 1599 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:38.674324 1599 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:38.681932 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:38.682772 1440 raft_consensus.cc:3032] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:38.683144 1440 raft_consensus.cc:729] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:38.684260 1440 consensus_queue.cc:260] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:38.685956 1440 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:38.687454 1599 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:38.687724 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:38.688419 2202 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:38.695686 2202 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:10:38.697286 1565 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:38.703323 3469 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:38.704331 3505 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:38.712358 3505 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20260808 01:10:38.710255 3469 raft_consensus.cc:686] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:38.730643 1440 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:10:38.733336 3469 consensus_queue.cc:237] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:38.754048 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 46211.3us
I20260808 01:10:38.804083 3467 raft_consensus.cc:979] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:38.804632 3467 raft_consensus.cc:1067] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:38.814220 3573 raft_consensus.cc:2932] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:38.818207 1524 raft_consensus.cc:2932] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:38.819183 1340 catalog_manager.cc:5495] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 3 to 4, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 4 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:38.824869 1447 raft_consensus.cc:2932] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:38.833314 1519 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58266
I20260808 01:10:38.834050 1519 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:38.834443 1519 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:10:38.849318 1390 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
I20260808 01:10:38.853945 1519 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:38.863529 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:38.862155 1340 catalog_manager.cc:5495] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index -1 to 4, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 3 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:38.864257 1439 raft_consensus.cc:3032] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:38.864864 1439 raft_consensus.cc:729] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:38.864766 1519 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:38.865792 1439 consensus_queue.cc:260] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:38.867105 1439 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:38.873795 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:38.874511 1604 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:38.882489 1604 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:38.888911 1489 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:38.891508 3582 raft_consensus.cc:2781] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:38.892630 3582 raft_consensus.cc:686] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:38.895752 3582 consensus_queue.cc:237] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:38.892786 1439 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
W20260808 01:10:38.905725 1390 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d80a0f28c7c94d45b7983fc0b1386caf, TS a4862d0938c84fc4bd840e4c17acae78 -> TS 95b661b03f1d460b9657e61d4b0247cd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:10:38.906147 1390 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d80a0f28c7c94d45b7983fc0b1386caf, TS a4862d0938c84fc4bd840e4c17acae78 -> TS 95b661b03f1d460b9657e61d4b0247cd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:10:38.927917 1390 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c12c84f631ee4ca7b405e28317db1b8a, TS 6b09dbefd24b4d52b385c4dff5697dc5 -> TS 95b661b03f1d460b9657e61d4b0247cd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:10:38.928465 1390 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet c12c84f631ee4ca7b405e28317db1b8a, TS 6b09dbefd24b4d52b385c4dff5697dc5 -> TS 95b661b03f1d460b9657e61d4b0247cd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:38.955415 1439 raft_consensus.cc:1261] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:38.959580 3469 consensus_queue.cc:1035] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:10:38.963724 1340 catalog_manager.cc:5495] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 2 to 3, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
W20260808 01:10:39.043375 1390 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9ea0da159669492a90fc257d91f4ba5c, TS a4862d0938c84fc4bd840e4c17acae78 -> TS 95b661b03f1d460b9657e61d4b0247cd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:10:39.043743 1390 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 9ea0da159669492a90fc257d91f4ba5c, TS a4862d0938c84fc4bd840e4c17acae78 -> TS 95b661b03f1d460b9657e61d4b0247cd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:39.073709 2202 raft_consensus.cc:1261] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:39.075086 3469 consensus_queue.cc:1035] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:10:39.200129 1439 raft_consensus.cc:1261] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:10:39.210067 3578 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:10:39.214064 2200 raft_consensus.cc:1261] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:10:39.217550 3573 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:10:39.280797 3594 ts_tablet_manager.cc:927] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:39.414908 3594 tablet_copy_client.cc:316] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.67:35675
I20260808 01:10:39.435918 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:39.478811 1440 raft_consensus.cc:1261] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:39.485553 3583 consensus_queue.cc:1035] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:39.516722 1596 raft_consensus.cc:1261] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:39.532686 1613 tablet_copy_service.cc:140] P a4862d0938c84fc4bd840e4c17acae78: Received BeginTabletCopySession request for tablet 1355c134ca5f45b381fb2b12b1865adb from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:34182)
I20260808 01:10:39.533403 1613 tablet_copy_service.cc:161] P a4862d0938c84fc4bd840e4c17acae78: Beginning new tablet copy session on tablet 1355c134ca5f45b381fb2b12b1865adb from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:34182: session id = 95b661b03f1d460b9657e61d4b0247cd-1355c134ca5f45b381fb2b12b1865adb
I20260808 01:10:39.561570 3582 consensus_queue.cc:1035] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260808 01:10:39.600129 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 44704.4us
I20260808 01:10:39.614531 1525 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:39.615082 1525 raft_consensus.cc:595] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:39.681056 1613 tablet_copy_source_session.cc:205] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78: Tablet Copy: opened 0 blocks and 1 log segments
W20260808 01:10:39.733661 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 135376us
I20260808 01:10:39.755074 1604 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:39.757841 3594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1355c134ca5f45b381fb2b12b1865adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:39.755718 1604 raft_consensus.cc:595] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Received request to transfer leadership to TS 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:39.784691 3606 raft_consensus.cc:979] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:39.785195 3606 raft_consensus.cc:1067] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Signalling peer 3fb1a6380d27401a831cfe5f86a4cc9d to start an election
I20260808 01:10:39.788091 1440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:55042
I20260808 01:10:39.788767 1440 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:39.789191 1440 raft_consensus.cc:3037] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:39.794394 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 01:10:39.796478 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:39.797219 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:39.805572 1440 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:39.808786 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:39.809001 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:39.809444 1519 raft_consensus.cc:3032] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:39.809697 2200 raft_consensus.cc:3037] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:39.809784 1519 raft_consensus.cc:729] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:39.810777 1519 consensus_queue.cc:260] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:39.812101 1519 raft_consensus.cc:3037] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:39.835753 2200 raft_consensus.cc:2454] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:39.837219 1408 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:39.839207 3587 raft_consensus.cc:2781] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:39.839896 3587 raft_consensus.cc:686] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:39.843149 1519 raft_consensus.cc:2454] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:39.843473 3587 consensus_queue.cc:237] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:39.850847 1440 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:39.880475 3594 tablet_copy_client.cc:791] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:39.881186 3594 tablet_copy_client.cc:655] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:39.898526 1340 catalog_manager.cc:5495] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:39.914187 3594 tablet_copy_client.cc:531] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:39.935550 3594 tablet_bootstrap.cc:492] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:40.109680 3607 raft_consensus.cc:979] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:40.110214 3604 raft_consensus.cc:1067] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Signalling peer 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:10:40.112824 2200 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:49260
I20260808 01:10:40.113616 2200 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:40.114068 2200 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:40.164141 2200 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:40.166500 2200 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:40.169950 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:40.171160 1447 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:40.171602 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:40.172328 1596 raft_consensus.cc:3032] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:40.172658 1596 raft_consensus.cc:729] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:40.173516 1596 consensus_queue.cc:260] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:40.175012 1596 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:40.187574 1596 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:10:40.194686 2171 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:40.195665 3608 raft_consensus.cc:2781] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:40.196177 3608 raft_consensus.cc:686] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:40.197082 3608 consensus_queue.cc:237] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:40.206610 1447 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:10:40.212908 3594 tablet_bootstrap.cc:492] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:40.216687 3594 tablet_bootstrap.cc:492] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:40.217646 3594 ts_tablet_manager.cc:1397] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.286s user 0.029s sys 0.004s
I20260808 01:10:40.226011 3594 raft_consensus.cc:348] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:40.231292 3594 raft_consensus.cc:729] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:40.238968 3594 consensus_queue.cc:260] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:40.242192 3594 ts_tablet_manager.cc:1428] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.024s user 0.008s sys 0.001s
I20260808 01:10:40.247675 1613 tablet_copy_service.cc:342] P a4862d0938c84fc4bd840e4c17acae78: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-1355c134ca5f45b381fb2b12b1865adb received from {username='slave'} at 127.0.0.1:34182
I20260808 01:10:40.248170 1613 tablet_copy_service.cc:433] P a4862d0938c84fc4bd840e4c17acae78: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-1355c134ca5f45b381fb2b12b1865adb on tablet 1355c134ca5f45b381fb2b12b1865adb with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:40.295434 1340 catalog_manager.cc:5495] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 3 to 4, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 4 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:40.344244 2200 raft_consensus.cc:1261] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:40.347410 3454 consensus_queue.cc:1035] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:40.410145 2202 raft_consensus.cc:1203] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[]
I20260808 01:10:40.448838 1519 raft_consensus.cc:1261] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:40.451161 3598 consensus_queue.cc:1035] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260808 01:10:40.477890 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 111082us
I20260808 01:10:40.684357 1447 raft_consensus.cc:1261] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:10:40.693940 3616 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:10:40.735095 1598 raft_consensus.cc:1261] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:10:40.769765 3616 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260808 01:10:40.941464 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 88240.5us
I20260808 01:10:41.038096 3573 raft_consensus.cc:1050] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:41.048965 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:41.059275 3573 consensus_queue.cc:237] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:41.159808 2201 raft_consensus.cc:1261] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEARNER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:10:41.163291 1447 raft_consensus.cc:1261] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:10:41.163612 1519 raft_consensus.cc:1261] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:10:41.161672 3573 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260808 01:10:41.192974 3634 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:41.200199 3632 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:41.246104 3573 raft_consensus.cc:2932] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:41.247674 1447 raft_consensus.cc:2932] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:41.258173 1524 raft_consensus.cc:2932] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:41.302711 2201 raft_consensus.cc:2932] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:41.313382 1338 catalog_manager.cc:5495] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index 4 to 5, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:10:41.353391 1447 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:10:41.355113 1447 raft_consensus.cc:595] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:41.361189 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260808 01:10:41.367303 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:41.371178 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:41.503211 1598 consensus_queue.cc:237] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:41.509654 3598 raft_consensus.cc:979] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:41.512430 3630 raft_consensus.cc:1067] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:41.514830 1518 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58266
I20260808 01:10:41.548364 2201 raft_consensus.cc:1261] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:41.547398 1524 raft_consensus.cc:1261] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:41.550499 3573 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:10:41.554991 3634 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:10:41.553503 1518 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:41.561658 1518 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:41.565264 3634 raft_consensus.cc:2932] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:41.590157 1328 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 1355c134ca5f45b381fb2b12b1865adb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:10:41.568476 1519 raft_consensus.cc:2932] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:41.607373 2201 raft_consensus.cc:2932] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:41.633324 1518 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:41.636438 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:41.637239 1439 raft_consensus.cc:3032] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:41.637584 1439 raft_consensus.cc:729] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:41.638861 1439 consensus_queue.cc:260] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:41.640125 1439 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:41.641865 1342 catalog_manager.cc:5495] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 5 to 6, VOTER 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) evicted. New cstate: current_term: 3 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:41.646730 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:41.647537 1596 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:41.655160 1518 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:41.664304 1429 tablet_service.cc:1514] Processing DeleteTablet for tablet 1355c134ca5f45b381fb2b12b1865adb with delete_type TABLET_DATA_TOMBSTONED (TS 3fb1a6380d27401a831cfe5f86a4cc9d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55036
I20260808 01:10:41.669847 1439 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:41.672108 1490 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:41.674600 3601 raft_consensus.cc:2781] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:41.675107 3601 raft_consensus.cc:686] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:41.676036 3601 consensus_queue.cc:237] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:41.693339 1596 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:41.724910 3643 tablet_replica.cc:330] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d: stopping tablet replica
I20260808 01:10:41.726193 3643 raft_consensus.cc:2229] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:10:41.727453 3643 raft_consensus.cc:2258] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:10:41.734468 3643 ts_tablet_manager.cc:1905] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:10:41.775372 1340 catalog_manager.cc:5495] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 3 to 4, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 4 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:41.787468 3643 ts_tablet_manager.cc:1918] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260808 01:10:41.791051 3643 log.cc:1196] T 1355c134ca5f45b381fb2b12b1865adb P 3fb1a6380d27401a831cfe5f86a4cc9d: Deleting WAL directory at /tmp/dist-test-taski9EQrI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786151308190403-32593-0/minicluster-data/ts-0-root/wals/1355c134ca5f45b381fb2b12b1865adb
I20260808 01:10:41.793802 1329 catalog_manager.cc:4841] TS 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): tablet 1355c134ca5f45b381fb2b12b1865adb (table test-workload [id=90f6392c55104263af6d6d1c9f7080de]) successfully deleted
I20260808 01:10:42.215711 1596 raft_consensus.cc:1261] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:10:42.219414 3606 consensus_queue.cc:1035] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:42.248773 1447 raft_consensus.cc:1261] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:10:42.251356 3606 consensus_queue.cc:1035] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260808 01:10:42.329035 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 66301.4us
W20260808 01:10:42.585316 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 57492.5us
W20260808 01:10:42.614730 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 88729.7us
I20260808 01:10:42.616389 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:42.767124 1518 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:10:42.768095 1518 raft_consensus.cc:595] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:42.777635 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260808 01:10:42.779603 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:42.780371 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260808 01:10:42.777101 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 44732.8us
W20260808 01:10:42.796504 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 47752.5us
I20260808 01:10:42.889392 1599 consensus_queue.cc:237] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:10:42.909286 1565 consensus_peers.cc:487] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 704a8e4165cc49079ada0c7993c21187. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:42.912144 1519 raft_consensus.cc:1261] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:10:42.912804 1439 raft_consensus.cc:1261] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:10:42.913805 3573 consensus_queue.cc:1035] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:42.914887 3634 consensus_queue.cc:1035] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:42.940608 3573 raft_consensus.cc:2932] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:43.023350 1439 raft_consensus.cc:2932] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
W20260808 01:10:43.087996 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 48625.7us
I20260808 01:10:43.098289 1340 catalog_manager.cc:5495] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index -1 to 4, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 4 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:43.089205 1519 raft_consensus.cc:2932] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:43.115052 1517 consensus_queue.cc:237] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:10:43.146020 1487 consensus_peers.cc:487] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a83a963773094d6db6495b7009fd8690. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:43.154692 1599 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260808 01:10:43.163555 3651 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:43.182382 3606 raft_consensus.cc:979] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5: : Instructing follower a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:43.182932 3606 raft_consensus.cc:1067] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Signalling peer a4862d0938c84fc4bd840e4c17acae78 to start an election
I20260808 01:10:43.192662 3631 raft_consensus.cc:2932] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:43.221040 1604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:45638
I20260808 01:10:43.220679 1599 raft_consensus.cc:2932] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:43.223181 1604 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:43.228248 1604 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:43.219651 1439 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260808 01:10:43.238562 3651 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:43.254030 1604 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:43.266309 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:43.266975 1517 raft_consensus.cc:3032] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:43.267273 1517 raft_consensus.cc:729] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:43.265273 1604 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:43.268407 1517 consensus_queue.cc:260] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:43.271447 1517 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:43.273586 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:43.274291 1439 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:43.285784 1517 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:10:43.286702 1440 raft_consensus.cc:2932] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:43.294852 1439 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:10:43.296419 1568 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:43.297327 3668 raft_consensus.cc:2781] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:43.297844 3668 raft_consensus.cc:686] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:43.298691 3668 consensus_queue.cc:237] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:43.309044 1517 consensus_queue.cc:237] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:10:43.344579 1487 consensus_peers.cc:487] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d80a0f28c7c94d45b7983fc0b1386caf. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:43.344873 1599 raft_consensus.cc:1261] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 01:10:43.348332 3651 consensus_queue.cc:1035] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:43.382352 1340 catalog_manager.cc:5495] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: config changed from index -1 to 5, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 5 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:43.416815 1440 raft_consensus.cc:1261] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 01:10:43.442216 1342 catalog_manager.cc:5495] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 3 to 4, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 4 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:10:43.468664 3631 consensus_queue.cc:1035] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:43.508522 3681 ts_tablet_manager.cc:927] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:43.511739 3651 raft_consensus.cc:2932] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:43.513217 1596 raft_consensus.cc:2932] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:43.522472 3681 tablet_copy_client.cc:316] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.67:35675
I20260808 01:10:43.528309 1613 tablet_copy_service.cc:140] P a4862d0938c84fc4bd840e4c17acae78: Received BeginTabletCopySession request for tablet 704a8e4165cc49079ada0c7993c21187 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:34182)
I20260808 01:10:43.529111 1613 tablet_copy_service.cc:161] P a4862d0938c84fc4bd840e4c17acae78: Beginning new tablet copy session on tablet 704a8e4165cc49079ada0c7993c21187 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:34182: session id = 95b661b03f1d460b9657e61d4b0247cd-704a8e4165cc49079ada0c7993c21187
I20260808 01:10:43.573109 1342 catalog_manager.cc:5495] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index -1 to 4, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
W20260808 01:10:43.609982 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 56997.8us
I20260808 01:10:43.608582 1613 tablet_copy_source_session.cc:205] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:43.654109 1440 raft_consensus.cc:2932] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
W20260808 01:10:43.667544 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 75901.1us
I20260808 01:10:43.675642 1596 consensus_queue.cc:237] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:43.681857 3681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 704a8e4165cc49079ada0c7993c21187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:43.686306 3684 ts_tablet_manager.cc:927] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:10:43.734273 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 67675.9us
W20260808 01:10:43.771658 1565 consensus_peers.cc:487] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c12c84f631ee4ca7b405e28317db1b8a. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:43.805739 3681 tablet_copy_client.cc:791] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:43.806476 3681 tablet_copy_client.cc:655] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
W20260808 01:10:43.812209 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 48461us
I20260808 01:10:43.821825 1517 raft_consensus.cc:1261] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260808 01:10:43.828197 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 71134.9us
I20260808 01:10:43.831789 1440 raft_consensus.cc:1261] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:10:43.816392 3684 tablet_copy_client.cc:316] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.66:37093
I20260808 01:10:43.834785 1534 tablet_copy_service.cc:140] P 6b09dbefd24b4d52b385c4dff5697dc5: Received BeginTabletCopySession request for tablet a83a963773094d6db6495b7009fd8690 from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:36560)
I20260808 01:10:43.835366 1534 tablet_copy_service.cc:161] P 6b09dbefd24b4d52b385c4dff5697dc5: Beginning new tablet copy session on tablet a83a963773094d6db6495b7009fd8690 from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:36560: session id = 95b661b03f1d460b9657e61d4b0247cd-a83a963773094d6db6495b7009fd8690
I20260808 01:10:43.845041 3700 consensus_queue.cc:1035] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:43.848228 3681 tablet_copy_client.cc:531] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:43.856616 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:43.854974 1534 tablet_copy_source_session.cc:205] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:43.869448 3668 consensus_queue.cc:1035] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:10:43.904651 1519 raft_consensus.cc:1261] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:10:43.914825 3667 consensus_queue.cc:1035] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260808 01:10:43.926399 1440 raft_consensus.cc:1261] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:10:44.011533 3684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83a963773094d6db6495b7009fd8690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:44.015619 3706 ts_tablet_manager.cc:927] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:44.015475 3668 consensus_queue.cc:1035] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.082s
I20260808 01:10:44.046837 3688 raft_consensus.cc:2932] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:44.055805 3681 tablet_bootstrap.cc:492] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:44.062553 1519 raft_consensus.cc:2932] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:44.100920 1440 raft_consensus.cc:2932] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:10:44.128439 3706 tablet_copy_client.cc:316] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.66:37093
W20260808 01:10:44.159194 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 48996.8us
I20260808 01:10:44.164140 1534 tablet_copy_service.cc:140] P 6b09dbefd24b4d52b385c4dff5697dc5: Received BeginTabletCopySession request for tablet d80a0f28c7c94d45b7983fc0b1386caf from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:36560)
I20260808 01:10:44.164752 1534 tablet_copy_service.cc:161] P 6b09dbefd24b4d52b385c4dff5697dc5: Beginning new tablet copy session on tablet d80a0f28c7c94d45b7983fc0b1386caf from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:36560: session id = 95b661b03f1d460b9657e61d4b0247cd-d80a0f28c7c94d45b7983fc0b1386caf
I20260808 01:10:44.166899 1596 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:44.167521 1596 raft_consensus.cc:595] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:10:44.224890 3710 log.cc:924] Time spent T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20260808 01:10:44.226794 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260808 01:10:44.227820 1534 tablet_copy_source_session.cc:205] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5: Tablet Copy: opened 0 blocks and 1 log segments
W20260808 01:10:44.230367 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 50067.2us
I20260808 01:10:44.241732 3706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d80a0f28c7c94d45b7983fc0b1386caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:44.245030 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:44.246024 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:44.256755 1340 catalog_manager.cc:5495] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: config changed from index -1 to 3, NON_VOTER 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) added. New cstate: current_term: 2 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260808 01:10:44.301964 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 67156.1us
W20260808 01:10:44.330025 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 205136us
W20260808 01:10:44.333294 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 68822.8us
I20260808 01:10:44.411895 3667 raft_consensus.cc:979] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:44.414168 3634 raft_consensus.cc:1067] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:10:44.403020 3684 tablet_copy_client.cc:791] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:44.420393 3684 tablet_copy_client.cc:655] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:44.430770 1519 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58280
I20260808 01:10:44.431423 1519 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:10:44.431810 1519 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:44.443171 1519 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:44.447547 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:44.448375 1594 raft_consensus.cc:3032] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:44.448722 1594 raft_consensus.cc:729] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:44.449542 1594 consensus_queue.cc:260] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:44.450779 1594 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:44.451426 3722 ts_tablet_manager.cc:927] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:44.451650 1519 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:44.905495 3684 tablet_copy_client.cc:531] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260808 01:10:45.258371 3706 consensus_meta.cc:287] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.822s user 0.011s sys 0.004s
I20260808 01:10:45.259948 3706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:49268 (request call id 895) took 1331 ms. Trace:
I20260808 01:10:45.261827 3706 rpcz_store.cc:276] 0808 01:10:43.928844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:43.931403 (+ 2559us) service_pool.cc:227] Handling call
0808 01:10:44.022213 (+ 90810us) ts_tablet_manager.cc:928] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
0808 01:10:44.240037 (+217824us) tablet_copy_client.cc:361] Tablet copy session begun
0808 01:10:44.382017 (+141980us) tablet_metadata.cc:641] Metadata flushed
0808 01:10:44.404643 (+ 22626us) tablet_copy_client.cc:490] Wrote new tablet metadata
0808 01:10:45.259043 (+854400us) tablet_copy_client.cc:411] Wrote new consensus metadata
0808 01:10:45.259410 (+ 367us) tablet_replica.cc:205] Creating consensus instance
0808 01:10:45.259765 (+ 355us) ts_tablet_manager.cc:983] Replica registered.
0808 01:10:45.259860 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":33912,"tablet-copy.queue_time_us":83600,"thread_start_us":160173,"threads_started":2}
I20260808 01:10:45.266252 3706 tablet_copy_client.cc:791] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:45.268532 3706 tablet_copy_client.cc:655] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:45.292515 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:45.293339 2201 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:10:45.295130 3681 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.842s user 0.007s sys 0.001s
W20260808 01:10:45.295544 3681 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.842s user 0.007s sys 0.001s
I20260808 01:10:45.297214 3684 tablet_bootstrap.cc:492] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:45.304024 3725 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:45.304697 3725 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.307905 3725 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:10:45.332436 1594 consensus_meta.cc:287] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.881s user 0.005s sys 0.010s
I20260808 01:10:45.333160 3725 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:45.334935 3725 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.338704 3725 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:45.323626 2201 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:45.340397 1594 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:45.344396 3722 tablet_copy_client.cc:316] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Beginning tablet copy session from remote peer at address 127.31.212.67:35675
I20260808 01:10:45.440176 3727 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:45.441044 3727 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.463021 3727 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:45.509411 3727 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:45.552248 3724 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:45.600265 3724 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.564297 3699 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:45.569818 3726 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:45.604583 3726 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.607283 3726 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:45.571506 3697 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:45.572846 3720 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:45.577148 3689 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:45.608132 3697 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.608150 3720 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.610595 3697 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:45.611492 3720 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:45.604583 3699 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.680637 3699 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:45.608470 3689 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.683485 3689 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:45.752367 3689 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:45.753295 3689 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.756292 3689 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:45.603597 3724 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:45.855613 3724 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:45.856447 3724 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.860338 3689 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:45.861106 3689 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.863750 3689 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:45.863984 3725 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:45.865044 3689 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:45.864836 3725 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.865935 3689 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.868750 3689 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:45.859656 3724 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:45.889473 3724 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:45.890388 3724 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.893415 3724 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:45.625286 3727 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.904891 3727 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:45.929023 3681 tablet_bootstrap.cc:492] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=1 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:45.930428 3681 tablet_bootstrap.cc:492] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:45.931557 3681 ts_tablet_manager.cc:1397] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 1.876s user 0.029s sys 0.055s
I20260808 01:10:45.932957 3724 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:45.934082 3720 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:45.933951 3724 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.933131 3689 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:45.934530 3697 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:45.934803 3720 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.867609 3725 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:45.937458 3725 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:45.937902 3720 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:45.935269 3697 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.935180 3689 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.938019 3725 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.940135 3689 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:45.941084 3734 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:45.936115 3699 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:45.937937 3724 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:45.942387 3697 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:45.942824 3699 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.944715 3735 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
W20260808 01:10:45.945348 3732 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.001s sys 0.000s
I20260808 01:10:45.946198 3699 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:45.940161 3725 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:45.946550 3681 raft_consensus.cc:348] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:45.949230 3684 tablet_bootstrap.cc:700] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd: Time spent opening tablet: real 0.606s user 0.004s sys 0.000s
I20260808 01:10:45.949318 3681 raft_consensus.cc:729] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:45.942791 3728 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:45.941673 3734 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.949584 3736 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:10:45.948446 3732 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.604s user 0.001s sys 0.000s
I20260808 01:10:45.950265 3728 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.951754 3735 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.950452 3681 consensus_queue.cc:260] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:45.954655 3725 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:45.972263 3725 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.961705 3735 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:45.974509 3735 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:45.955950 3736 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.957422 3728 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:45.960428 3681 ts_tablet_manager.cc:1428] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.028s user 0.005s sys 0.002s
I20260808 01:10:45.955152 3734 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:45.975615 3735 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:45.978200 3736 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:45.978972 3735 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:45.979576 3734 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:45.979709 3735 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:45.980149 3725 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:45.980327 3734 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:45.980491 3735 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:45.983028 3734 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:45.985277 3735 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.012600 3684 tablet_bootstrap.cc:492] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:46.012895 3740 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:46.014170 3739 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:46.013833 3740 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.013862 3684 tablet_bootstrap.cc:492] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:46.017197 3684 ts_tablet_manager.cc:1397] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 0.720s user 0.035s sys 0.011s
I20260808 01:10:46.017549 3740 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:46.017562 1408 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.647s user 0.009s sys 0.013s
I20260808 01:10:46.014732 3739 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.021171 3684 raft_consensus.cc:348] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:46.022514 3684 raft_consensus.cc:729] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
W20260808 01:10:46.022956 1408 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.653s user 0.009s sys 0.016s
I20260808 01:10:46.023361 3684 consensus_queue.cc:260] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:46.026454 3684 ts_tablet_manager.cc:1428] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260808 01:10:46.027819 3739 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:46.049134 3739 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:46.049881 3739 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.052186 3739 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.052819 3735 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:46.057461 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.058035 3735 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:46.053879 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.055650 3728 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:46.056411 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.060817 3735 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.059330 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.061678 1445 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 1.
I20260808 01:10:46.061830 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.060316 3728 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.057217 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.064555 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.065251 1440 raft_consensus.cc:2454] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:10:46.067350 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.069260 1442 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:46.067041 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.070533 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.070587 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.070779 1440 raft_consensus.cc:2454] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 1.
I20260808 01:10:46.071202 1447 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:10:46.072255 3741 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:46.073181 3728 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.076812 3728 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.077333 3728 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.077998 3735 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.078204 3744 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.078907 3744 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:46.078768 3735 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:46.081409 3735 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.081534 3744 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.079586 3728 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.065977 3743 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:46.089733 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.064065 3739 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:46.073015 3741 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.095074 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.096181 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.092934 3739 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.098078 3745 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.098667 3745 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:46.102097 3742 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:46.088528 3743 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.109197 3747 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:46.109606 3739 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.116288 3728 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:46.116683 3743 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.117501 3743 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:46.117352 3728 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:46.118093 3743 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.120075 3747 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.121315 3728 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.124212 3739 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:46.125113 3739 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.126866 3747 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.102689 3742 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.134431 3744 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.137605 3744 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.126477 3740 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:46.143044 3744 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.135597 3728 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:46.140069 3735 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:46.144853 3728 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.145009 3735 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.147009 3728 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.147532 3735 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.141985 3740 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.150481 3740 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:46.157236 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.158070 1443 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:10:46.169440 3741 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.170418 3745 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.184558 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.186782 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.159811 3743 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.190600 3747 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.192953 3747 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.195032 3747 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.260232 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.247958 3742 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:46.190716 3750 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:46.250833 3745 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:46.289271 3754 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:46.289891 3754 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.296137 3749 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:46.335870 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.296746 3749 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.337872 3755 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:46.337203 1598 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:46.341972 3756 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:46.342051 1613 tablet_copy_service.cc:140] P a4862d0938c84fc4bd840e4c17acae78: Received BeginTabletCopySession request for tablet c12c84f631ee4ca7b405e28317db1b8a from peer 95b661b03f1d460b9657e61d4b0247cd ({username='slave'} at 127.0.0.1:34182)
I20260808 01:10:46.329493 3745 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.334234 3740 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:46.345377 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.349711 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.338402 3755 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.344213 3742 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:46.342959 1613 tablet_copy_service.cc:161] P a4862d0938c84fc4bd840e4c17acae78: Beginning new tablet copy session on tablet c12c84f631ee4ca7b405e28317db1b8a from peer 95b661b03f1d460b9657e61d4b0247cd at {username='slave'} at 127.0.0.1:34182: session id = 95b661b03f1d460b9657e61d4b0247cd-c12c84f631ee4ca7b405e28317db1b8a
I20260808 01:10:46.353188 1598 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:46.356930 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.342720 3756 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:46.357492 3741 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.350767 3747 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.358421 3754 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.358160 3751 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:46.283977 3750 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.357652 3742 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.367043 3743 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:46.366914 3759 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:46.367624 3743 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.370227 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.370765 3750 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.370965 1604 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:46.371428 3761 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.357213 3739 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:46.347366 3740 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.475188 1598 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:10:46.364609 3751 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.500881 1596 raft_consensus.cc:2454] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 1.
I20260808 01:10:46.366909 3754 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.536684 3743 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.361661 3741 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.365224 3747 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.508059 3740 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.522251 3749 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.660003 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.670652 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.572508 3743 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:46.493584 3759 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.688265 3759 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.372982 3761 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:46.690829 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.685648 3751 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.681599 3742 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.660332 3755 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:46.573880 3754 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.660720 3749 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.710757 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.712025 1598 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 1.
I20260808 01:10:46.716279 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.694310 3759 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.705598 3745 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.736229 3745 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:46.726060 3757 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:46.737295 3757 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.731331 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.740335 3745 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.741644 1598 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:46.742859 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.743108 1519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1751) took 2292 ms. Trace:
I20260808 01:10:46.743450 1519 rpcz_store.cc:276] 0808 01:10:44.452698 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:46.742289 (+2289591us) service_pool.cc:227] Handling call
0808 01:10:46.742623 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:46.742796 (+ 173us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:46.742822 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:46.742862 (+ 40us) raft_consensus.cc:1631] Marking committed up to 1
0808 01:10:46.742893 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:46.742950 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:46.743054 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:46.750137 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.752619 3757 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.753623 3757 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.754366 3757 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.756546 3747 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.780639 3747 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.781260 3747 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.781701 3757 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.729235 1614 tablet_copy_service.cc:342] P a4862d0938c84fc4bd840e4c17acae78: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-704a8e4165cc49079ada0c7993c21187 received from {username='slave'} at 127.0.0.1:34182
I20260808 01:10:46.729915 3754 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.707532 3756 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:46.732321 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.733242 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.734166 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.721875 3762 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.727628 3761 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.743414 3745 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.757583 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.798596 1447 raft_consensus.cc:2454] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
W20260808 01:10:46.761365 1487 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.704s (SENT) [suppressed 1 similar messages]
W20260808 01:10:46.764357 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 100228us
I20260808 01:10:46.766378 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.766582 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.767663 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.772341 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:46.692255 3743 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.706514 3758 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:46.792133 1596 raft_consensus.cc:2454] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:46.794608 3762 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.802742 1445 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:46.800930 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.783154 3760 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:46.719815 3749 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.790447 1614 tablet_copy_service.cc:433] P a4862d0938c84fc4bd840e4c17acae78: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-704a8e4165cc49079ada0c7993c21187 on tablet 704a8e4165cc49079ada0c7993c21187 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:46.818743 3762 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:10:46.821161 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 59838.5us
W20260808 01:10:46.821081 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 108345us
I20260808 01:10:46.756829 3759 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.825320 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.827389 1517 raft_consensus.cc:2454] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:46.828907 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.830753 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.808118 3765 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:46.833192 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.837368 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:46.835578 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:46.837796 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:46.836462 1613 tablet_copy_source_session.cc:205] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:10:46.832036 3741 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.828609 3762 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.846231 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.846534 3762 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.847985 2205 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:10:46.849050 3762 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.844843 2207 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:46.850581 3761 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:46.851125 3761 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.851496 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:46.852706 3743 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.853256 3761 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.840473 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:46.840768 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:46.852854 2172 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:46.854056 2204 raft_consensus.cc:2454] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:46.841233 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.841312 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.844466 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.842839 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.853060 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:46.840402 2202 raft_consensus.cc:1203] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEARNER]: Deduplicated request from leader. Original: 5.4->[5.5-5.5] Dedup: 5.5->[]
I20260808 01:10:46.855895 1524 raft_consensus.cc:2454] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:46.857836 2172 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:46.860028 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.860711 1518 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:10:46.859000 1408 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:46.863729 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.861753 2172 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:46.855396 3697 raft_consensus.cc:2781] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:10:46.890475 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:46.828924 3760 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.892992 3728 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:10:46.893776 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.895171 1408 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:46.903172 1520 raft_consensus.cc:2454] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:10:46.855334 3743 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:46.902489 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.903635 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.904008 1519 raft_consensus.cc:2454] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:46.815477 3758 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.900604 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.905455 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.905776 3743 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.906127 1604 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:46.906941 3758 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.907119 1516 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:10:46.907768 3747 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.902161 2172 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:46.908628 1522 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:10:46.857640 3761 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:46.914830 3749 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.858937 3765 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:46.918799 2172 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:46.923240 1568 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:46.924350 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.906961 3728 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:46.925370 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.925793 3728 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:46.926276 3760 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:46.902514 3697 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:46.926594 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.926977 3697 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:46.927731 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.928704 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.935245 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:46.936237 1339 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37234 (request call id 234) took 2023 ms. Trace:
I20260808 01:10:46.936565 3766 raft_consensus.cc:2781] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:46.937103 3766 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:46.937503 3766 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:46.937927 3759 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.916391 3699 raft_consensus.cc:2781] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:46.938638 3699 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.939078 3699 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:46.940038 3689 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:10:46.909102 3767 raft_consensus.cc:2781] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:46.936625 1339 rpcz_store.cc:276] 0808 01:10:45.148544 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:46.926660 (+1778116us) service_pool.cc:227] Handling call
0808 01:10:46.929056 (+ 2396us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":0}
I20260808 01:10:46.941371 1340 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55758 (request call id 333) took 2832 ms. Trace:
I20260808 01:10:46.943456 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.922412 3771 raft_consensus.cc:2781] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:10:46.946928 3771 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:46.947360 3771 raft_consensus.cc:3037] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:46.944442 1525 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:46.940435 3689 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.941602 3767 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.951539 3689 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:46.952246 3767 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:46.960992 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:46.961943 2201 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:10:46.963909 3772 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:10:46.921258 3761 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:46.964171 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.964463 3772 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:46.964900 3772 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:46.947582 3743 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:46.963120 1568 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:10:46.968230 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 73139.2us
I20260808 01:10:46.968976 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.969259 1408 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:46.968423 2172 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:46.941768 1340 rpcz_store.cc:276] 0808 01:10:44.108819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:44.108990 (+ 171us) service_pool.cc:227] Handling call
0808 01:10:44.278354 (+169364us) spinlock_profiling.cc:243] Waited 2.59 ms on lock 0x7b5400002728. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe350386361 00007fe3504102cf 00007fe350411a21 00007fe352ac61f1 00007fe352ac6a16 00007fe352990d7b 00007fe352a944fe 00007fe350da5fb4 00007fe350da5f42 00007fe350da5e71 00007fe350da5dec
0808 01:10:46.941280 (+2662926us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 01:10:44.448270 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:10:44.448860 (+ 590us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:10:44.448915 (+ 55us) write_op.cc:435] Acquired schema lock
0808 01:10:44.448933 (+ 18us) tablet.cc:660] Decoding operations
0808 01:10:44.449302 (+ 369us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:10:44.449382 (+ 80us) write_op.cc:641] Partition lock acquired for write op
0808 01:10:44.449408 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
0808 01:10:44.449609 (+ 201us) tablet.cc:699] Row locks acquired
0808 01:10:44.449628 (+ 19us) write_op.cc:260] PREPARE: finished
0808 01:10:44.449985 (+ 357us) write_op.cc:270] Start()
0808 01:10:44.450103 (+ 118us) write_op.cc:276] Timestamp: P: 1786151444449917 usec, L: 0
0808 01:10:44.450124 (+ 21us) op_driver.cc:331] REPLICATION: starting
0808 01:10:45.244595 (+794471us) log.cc:842] Serialized 599 byte log entry
0808 01:10:45.294048 (+ 49453us) spinlock_profiling.cc:243] Waited 732 ms on lock 0x7b5400002dd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3b198 00007fe35044eade 00007fe35044e2b8 00007fe350452ba1 00007fe350452b59 00007fe350452ae9 00007fe350452ab1 00007fe350451dad 00007fe352d7b204 00007fe352d7b039
0808 01:10:45.344505 (+ 50457us) op_driver.cc:444] REPLICATION: finished
0808 01:10:45.915954 (+571449us) write_op.cc:301] APPLY: starting
0808 01:10:45.916057 (+ 103us) tablet.cc:1365] starting BulkCheckPresence
0808 01:10:45.916162 (+ 105us) tablet.cc:1368] finished BulkCheckPresence
0808 01:10:45.916185 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 01:10:45.919802 (+ 3617us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 01:10:45.948080 (+ 28278us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 01:10:45.948963 (+ 883us) write_op.cc:312] APPLY: finished
0808 01:10:45.952767 (+ 3804us) log.cc:842] Serialized 23 byte log entry
0808 01:10:45.953641 (+ 874us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:10:45.953733 (+ 92us) write_op.cc:454] Released schema lock
0808 01:10:45.953777 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0808 01:10:45.954113 (+ 336us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:10:45.954141 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":130900,"reported_tablets":2,"spinlock_wait_cycles":5957888,"child_traces":[["op",{"apply.queue_time_us":571120,"apply.run_cpu_time_us":5473,"apply.run_wall_time_us":38435,"num_ops":1,"prepare.queue_time_us":121029,"prepare.run_cpu_time_us":13775,"prepare.run_wall_time_us":889973,"raft.queue_time_us":53480,"raft.run_cpu_time_us":1888,"raft.run_wall_time_us":611228,"replication_time_us":894312,"spinlock_wait_cycles":1683692416,"thread_start_us":752330,"threads_started":4,"wal-append.queue_time_us":40893}]]}
I20260808 01:10:46.971535 3742 raft_consensus.cc:2781] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:46.972126 3742 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:46.972479 3742 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:46.975777 3718 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:10:46.976388 3718 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:46.976781 3718 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:46.982903 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:46.984110 1525 raft_consensus.cc:2454] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:46.987663 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.990279 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:46.990512 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:46.994457 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.018002 3774 raft_consensus.cc:2781] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:47.018558 3774 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:47.018949 3774 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:47.031240 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.041059 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:47.072964 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:47.074409 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.075223 1524 raft_consensus.cc:2454] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 1.
I20260808 01:10:47.073832 2201 raft_consensus.cc:2454] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
I20260808 01:10:47.088913 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.128224 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:47.135511 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:10:47.088423 1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800232d40 after lost signal to thread 32596
I20260808 01:10:47.139988 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.147424 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.153613 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.155059 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.164467 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.165437 1518 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:10:47.173534 3764 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:10:47.173622 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 91385.4us
I20260808 01:10:47.174597 3764 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:47.174571 3780 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:47.181418 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.183372 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.142829 3697 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:47.205164 3718 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:47.206260 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 91373.3us
W20260808 01:10:47.208053 1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800233180 after lost signal to thread 1376
W20260808 01:10:47.215013 1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800093c60 after lost signal to thread 1473
W20260808 01:10:47.215294 1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800143080 after lost signal to thread 1474
I20260808 01:10:47.207520 3755 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:47.187810 3764 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:47.216337 1626 debug-util.cc:398] Leaking SignalData structure 0x7b08001e49e0 after lost signal to thread 1551
I20260808 01:10:47.213197 3728 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:47.211217 3689 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:47.187942 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.228646 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.262017 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.261075 3777 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
W20260808 01:10:47.273954 1626 debug-util.cc:398] Leaking SignalData structure 0x7b080036f500 after lost signal to thread 1553
I20260808 01:10:47.288252 1516 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:47.290441 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:10:47.291080 1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800363280 after lost signal to thread 1629
W20260808 01:10:47.291416 1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800311980 after lost signal to thread 1630
I20260808 01:10:47.184000 3699 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:47.315191 3728 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:47.315418 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.316035 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.316385 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.316751 1523 raft_consensus.cc:2454] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 1.
I20260808 01:10:47.316872 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.317564 1520 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 1.
I20260808 01:10:47.318747 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.320091 1534 tablet_copy_service.cc:342] P 6b09dbefd24b4d52b385c4dff5697dc5: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-a83a963773094d6db6495b7009fd8690 received from {username='slave'} at 127.0.0.1:36560
I20260808 01:10:47.320573 1534 tablet_copy_service.cc:433] P 6b09dbefd24b4d52b385c4dff5697dc5: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-a83a963773094d6db6495b7009fd8690 on tablet a83a963773094d6db6495b7009fd8690 with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:47.321892 3767 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:47.323096 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.326169 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.343603 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.279897 3697 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:10:47.358338 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 144636us
I20260808 01:10:47.281688 3765 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:47.236348 3772 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:47.250799 3771 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:47.362213 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.369345 1604 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:47.362525 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.370555 3776 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:47.371268 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:47.371143 3776 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:47.373296 3776 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:47.373934 3776 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:47.374009 3767 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:47.232323 3755 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:47.374493 3776 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:47.276928 3774 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:47.376442 3699 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:47.377707 3774 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:47.251668 3766 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:47.379693 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.364125 2171 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:47.187471 3780 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:47.361428 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 107595us
I20260808 01:10:47.357908 3775 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:47.382728 1596 raft_consensus.cc:1226] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:47.387100 2171 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:47.287810 3777 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:47.366622 3689 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:47.398437 3780 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:47.386449 3775 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:47.401438 3775 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:47.402271 3718 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:47.404322 3772 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:47.404189 3742 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:47.405411 3718 raft_consensus.cc:2781] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:47.405813 3718 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:47.406178 3718 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:47.406482 3742 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:47.409960 2171 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:47.411540 2171 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:47.415244 2171 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:47.419327 2171 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:47.386067 3779 raft_consensus.cc:2781] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:47.422382 2171 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:10:47.425221 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 58764.6us
I20260808 01:10:47.426262 2171 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:47.426417 1410 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:47.420945 3786 raft_consensus.cc:2781] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:10:47.428494 3786 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:47.428870 3786 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:10:47.429760 1626 debug-util.cc:398] Leaking SignalData structure 0x7b080030d120 after lost signal to thread 2233
I20260808 01:10:47.430032 3755 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:10:47.430343 1626 debug-util.cc:398] Leaking SignalData structure 0x7b08002be4e0 after lost signal to thread 2234
I20260808 01:10:47.424355 3699 raft_consensus.cc:2781] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:10:47.431137 3699 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:47.431521 3699 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:47.421648 3785 raft_consensus.cc:2781] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:10:47.432225 3785 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:47.387238 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:47.432616 3785 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:10:47.430830 1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800356460 after lost signal to thread 3681
W20260808 01:10:47.433928 1626 debug-util.cc:398] Leaking SignalData structure 0x7b0800356420 after lost signal to thread 3684
W20260808 01:10:47.434309 1626 debug-util.cc:398] Leaking SignalData structure 0x7b08002d8c20 after lost signal to thread 3708
I20260808 01:10:47.417351 3689 raft_consensus.cc:2781] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 01:10:47.436053 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 183866us
I20260808 01:10:47.432636 1338 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55752 (request call id 305) took 3070 ms. Trace:
I20260808 01:10:47.421566 3779 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:10:47.434631 1626 debug-util.cc:398] Leaking SignalData structure 0x7b080025ece0 after lost signal to thread 3722
I20260808 01:10:47.440559 1338 rpcz_store.cc:276] 0808 01:10:44.362280 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:44.363093 (+ 813us) service_pool.cc:227] Handling call
0808 01:10:47.432556 (+3069463us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260808 01:10:47.433516 3767 raft_consensus.cc:2781] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:47.434598 1410 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:47.446286 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:47.448704 2201 raft_consensus.cc:2379] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 5: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:10:47.450994 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:47.451869 1439 raft_consensus.cc:2379] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 4: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:47.438866 3689 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:47.451974 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:47.447744 1410 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:47.435745 3772 raft_consensus.cc:2781] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:10:47.491705 2206 raft_consensus.cc:2379] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 4: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:10:47.441805 3779 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:47.452440 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:47.493026 1447 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:47.494241 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:47.498160 1604 raft_consensus.cc:1226] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:47.500142 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:47.444972 3697 raft_consensus.cc:2781] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:47.452617 3735 raft_consensus.cc:2781] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:47.504575 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:47.443900 3776 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:47.507388 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.508421 1410 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:47.509328 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:47.444222 3767 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:47.513021 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.514577 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:47.514688 1410 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:47.515444 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.513873 1596 raft_consensus.cc:2454] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:47.519428 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.489753 3689 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:47.520133 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.520260 1516 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:10:47.519340 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.490765 3775 raft_consensus.cc:2726] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:47.527480 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.491106 3772 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:47.498865 3777 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:47.552796 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:47.553581 1440 raft_consensus.cc:3037] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:47.511966 3735 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:47.556147 1445 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:47.556357 3735 raft_consensus.cc:3037] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:47.510083 3697 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:47.559610 3761 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:47.564632 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 50009.5us
I20260808 01:10:47.516671 3767 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:47.525887 3776 raft_consensus.cc:2781] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:10:47.529047 3766 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:47.532241 3772 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:47.530728 3790 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:47.584475 3790 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:47.585469 3722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12c84f631ee4ca7b405e28317db1b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:10:47.586024 1410 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
W20260808 01:10:47.587484 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 82245.9us
I20260808 01:10:47.588431 1447 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:10:47.589066 1487 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:47.590688 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.590790 1487 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:47.592401 1440 raft_consensus.cc:2454] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:47.567885 3697 raft_consensus.cc:3037] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:47.566893 3789 raft_consensus.cc:2781] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:47.556223 3771 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:47.608911 1342 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55756 (request call id 335) took 3307 ms. Trace:
I20260808 01:10:47.574671 3776 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:10:47.610376 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 50004.5us
I20260808 01:10:47.612411 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.609468 1342 rpcz_store.cc:276] 0808 01:10:44.301842 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:44.302011 (+ 169us) service_pool.cc:227] Handling call
0808 01:10:47.608818 (+3306807us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260808 01:10:47.575459 3774 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:47.619046 1567 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:47.627424 1567 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:47.628940 1567 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:47.598075 3752 raft_consensus.cc:2781] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:10:47.598389 3769 raft_consensus.cc:2781] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:10:47.601076 3789 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:47.609519 3728 raft_consensus.cc:2781] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:10:47.663411 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.635540 3776 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:47.688575 1567 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:47.693104 1567 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:47.651304 3766 raft_consensus.cc:2781] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:10:47.651629 3752 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:47.694798 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:10:47.697513 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 90097us
I20260808 01:10:47.661630 3769 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:47.660772 3755 raft_consensus.cc:2781] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:10:47.702978 3755 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:47.664098 3791 raft_consensus.cc:2781] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:47.687465 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:47.700246 3752 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:47.665398 3789 raft_consensus.cc:3037] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:47.718158 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.665035 3728 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:47.651739 3774 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:47.699102 3766 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:47.703743 3769 raft_consensus.cc:3037] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:47.711979 3755 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:47.714753 3756 raft_consensus.cc:2781] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:10:47.720135 3790 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:47.724946 3728 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:47.735582 3766 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:47.707633 3791 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:47.739044 3756 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:47.764999 2171 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:47.714489 3795 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:10:47.770877 2171 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:47.771343 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.781203 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.782652 3706 tablet_copy_client.cc:531] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:47.788269 2172 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:47.789903 2172 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:47.770179 3756 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:47.767357 3791 raft_consensus.cc:3037] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:10:47.819921 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 113738us
W20260808 01:10:47.830008 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 57470.4us
I20260808 01:10:47.773911 3794 raft_consensus.cc:2726] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:47.771586 3795 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:47.818292 3783 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:47.774443 3778 raft_consensus.cc:2781] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:10:47.845151 3720 raft_consensus.cc:2781] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election won for term 3
W20260808 01:10:47.897193 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 107414us
W20260808 01:10:47.897233 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 56557.4us
I20260808 01:10:47.858565 3790 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:47.844761 3699 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:47.861394 3795 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:10:47.907469 2171 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:10:47.918684 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 225627us
I20260808 01:10:47.920344 3808 raft_consensus.cc:2726] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:47.912091 3720 raft_consensus.cc:686] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:47.923969 3748 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:47.924664 3774 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:47.935179 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:47.936831 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:47.931516 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.927147 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:47.874795 3778 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:47.906769 3794 raft_consensus.cc:2781] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:47.904701 3742 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:47.959756 3778 raft_consensus.cc:3037] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:47.994376 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:47.913313 3790 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:47.978988 3794 raft_consensus.cc:686] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:47.978744 3742 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:47.977046 3748 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:48.032792 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 326252us
I20260808 01:10:47.916510 3783 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.034503 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:48.039361 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 63392us
I20260808 01:10:48.043732 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:48.042907 1410 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:48.046223 3751 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:48.046957 1410 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:48.047037 3751 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.048029 3807 raft_consensus.cc:2726] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:10:48.049928 1410 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:48.050633 3807 raft_consensus.cc:2781] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:48.050989 3807 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:48.051333 3807 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:48.052058 3748 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:48.052686 1410 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:48.053846 3781 raft_consensus.cc:2726] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:10:48.054515 3783 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:48.056496 3751 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:48.057829 1410 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:48.060463 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:48.114933 3787 raft_consensus.cc:2726] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:10:48.060263 3774 raft_consensus.cc:2726] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:10:48.130868 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:48.131768 1600 raft_consensus.cc:2362] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4. Re-sending same reply.
I20260808 01:10:48.132236 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:48.026731 3720 consensus_queue.cc:237] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.147199 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:48.150816 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:48.036170 3794 consensus_queue.cc:237] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:48.166422 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 108655us
W20260808 01:10:48.167447 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 260055us
I20260808 01:10:48.168329 2171 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:48.170514 2171 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:48.172022 2171 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:48.175494 1410 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:10:48.176405 1445 consensus_meta.cc:287] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.620s user 0.003s sys 0.053s
I20260808 01:10:48.177361 1445 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:48.182518 2171 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:48.175387 3780 raft_consensus.cc:2726] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:48.183395 3799 raft_consensus.cc:2781] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:10:48.184011 3799 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:48.184044 3806 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:48.184458 3799 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:48.187697 1410 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:48.188769 2171 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:48.190948 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:48.194020 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:48.201661 3784 raft_consensus.cc:2781] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:10:48.201920 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:48.200030 2171 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:48.202160 3784 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:48.209708 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.178735 3699 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:48.209417 2171 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:48.212303 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:48.213248 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.187053 3802 raft_consensus.cc:2781] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:48.210870 3808 raft_consensus.cc:2726] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:10:48.218261 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.213876 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.219216 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:48.216464 2172 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:10:48.207319 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 79785us
I20260808 01:10:48.202636 3742 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:48.223765 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.223371 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.199347 3780 raft_consensus.cc:2781] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:10:48.223713 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.224568 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.226593 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.202961 3781 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:48.212755 3784 raft_consensus.cc:3037] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:10:48.220372 3786 consensus_meta.cc:287] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.791s user 0.026s sys 0.011s
I20260808 01:10:48.233188 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.233278 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.233577 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:48.235641 1438 raft_consensus.cc:1226] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:48.236501 1444 raft_consensus.cc:1226] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:48.236316 1489 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:48.236644 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.227754 2206 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:48.184759 3806 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.219224 3783 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:48.249886 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:48.227381 3802 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:48.227156 3780 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:10:48.249969 3718 consensus_meta.cc:287] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.843s user 0.019s sys 0.021s
I20260808 01:10:48.266301 2171 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:48.266402 1487 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:48.248857 3742 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:48.274789 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.278107 1489 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:48.257370 3790 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:48.279609 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.280598 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.281170 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.257625 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:48.301332 1446 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:48.256839 3797 raft_consensus.cc:2726] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:10:48.271278 3780 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:48.267139 3802 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:48.311255 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.311430 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.312004 1448 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:48.312386 1438 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:48.312939 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.313571 1448 raft_consensus.cc:2454] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:10:48.313861 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.316504 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.317490 1438 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:48.320716 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.321148 1438 raft_consensus.cc:1226] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:48.321832 3796 raft_consensus.cc:2781] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:48.322280 3796 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:48.322774 1438 raft_consensus.cc:1226] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:48.251250 3786 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.322139 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.314364 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.249747 3781 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:48.279150 3808 raft_consensus.cc:2726] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:48.328469 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:48.322727 3796 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:48.304368 3804 raft_consensus.cc:2726] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:10:48.267220 3764 consensus_queue.cc:1046] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260808 01:10:48.270975 3783 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.304105 3809 raft_consensus.cc:2781] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:10:48.352250 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:48.276136 3718 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.369275 1446 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:48.353037 1443 raft_consensus.cc:2454] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 1.
W20260808 01:10:48.352636 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 86283.8us
I20260808 01:10:48.356747 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.370524 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.286417 3742 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:48.370536 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 122475us
I20260808 01:10:48.360648 3765 raft_consensus.cc:2781] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 01:10:48.373696 3789 consensus_meta.cc:287] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.638s user 0.018s sys 0.008s
W20260808 01:10:48.377228 3689 consensus_meta.cc:287] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.824s user 0.006s sys 0.011s
I20260808 01:10:48.373111 3751 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:48.379053 3764 raft_consensus.cc:3032] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:48.381243 3809 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:48.400584 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.371150 3810 raft_consensus.cc:2726] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:10:48.391559 3752 consensus_meta.cc:287] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.667s user 0.006s sys 0.006s
I20260808 01:10:48.393293 3765 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:48.419763 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.402096 1603 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:48.397572 3800 raft_consensus.cc:2781] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:48.460014 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.460621 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.462198 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.462517 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.460090 2201 raft_consensus.cc:1226] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:48.466544 2201 raft_consensus.cc:1226] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 5. Current term is 6. Ops: []
I20260808 01:10:48.468297 3811 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:48.400141 2200 raft_consensus.cc:1226] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:48.416384 1438 raft_consensus.cc:1226] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:10:48.425161 3809 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:48.409457 3764 raft_consensus.cc:729] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
W20260808 01:10:48.485504 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 207452us
I20260808 01:10:48.394897 3814 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:48.488118 1489 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:48.495730 1489 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:48.456007 3765 raft_consensus.cc:3037] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:48.502434 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:10:48.479516 3728 consensus_meta.cc:287] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.719s user 0.002s sys 0.034s
I20260808 01:10:48.513341 3818 raft_consensus.cc:2726] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:10:48.518564 3697 consensus_meta.cc:287] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.900s user 0.027s sys 0.022s
I20260808 01:10:48.520888 3718 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:48.523087 2205 raft_consensus.cc:1226] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:48.391072 3807 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:48.399492 3689 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.418601 3751 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.484656 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.469216 3811 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:48.551577 3821 raft_consensus.cc:2726] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:48.525233 2205 raft_consensus.cc:1226] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:48.507274 3800 raft_consensus.cc:686] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:48.399294 3789 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:48.563875 1440 raft_consensus.cc:1226] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:48.567111 2206 raft_consensus.cc:1226] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:48.570674 3816 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:48.573017 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:48.594879 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:48.595139 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:48.596129 1604 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:10:48.599279 3800 consensus_queue.cc:237] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.519269 3697 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.634298 3765 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:48.646783 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:48.638443 3765 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:48.482167 3792 consensus_queue.cc:1046] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.310s
I20260808 01:10:48.659483 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.652170 2208 raft_consensus.cc:1226] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:48.648525 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:48.653404 3813 log.cc:924] Time spent T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20260808 01:10:48.653522 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:48.622265 3735 consensus_meta.cc:287] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 1.065s user 0.023s sys 0.017s
I20260808 01:10:48.533250 3728 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:48.436187 3752 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.654899 2203 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 2: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:10:48.655601 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:48.656458 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:48.539672 3806 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:48.506376 3814 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:48.653440 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:48.662489 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.560166 3781 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:48.659427 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:48.581504 3786 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:48.664481 2204 raft_consensus.cc:1226] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 2. Current term is 3. Ops: []
W20260808 01:10:48.668824 3776 consensus_meta.cc:287] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.971s user 0.010s sys 0.025s
I20260808 01:10:48.669807 3819 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:48.588475 3783 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:48.675274 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.676725 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:10:48.678134 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 182561us
I20260808 01:10:48.679214 1601 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:48.663585 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.684908 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.686303 3689 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:48.686697 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.687291 3728 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:48.688102 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:48.691021 3697 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:48.688156 1443 raft_consensus.cc:2379] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 2: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
W20260808 01:10:48.670971 3772 consensus_meta.cc:287] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 1.074s user 0.029s sys 0.027s
W20260808 01:10:48.676553 3785 consensus_meta.cc:287] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 1.240s user 0.017s sys 0.009s
I20260808 01:10:48.693439 3742 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:48.710664 2204 raft_consensus.cc:1226] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:48.572602 3764 consensus_queue.cc:260] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:48.714473 3814 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:48.716423 2208 raft_consensus.cc:1203] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEARNER]: Deduplicated request from leader. Original: 4.3->[4.4-4.4] Dedup: 4.4->[]
I20260808 01:10:48.717303 1445 raft_consensus.cc:1226] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:48.657639 3811 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:48.770653 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.742048 3706 tablet_bootstrap.cc:492] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
I20260808 01:10:48.742784 3785 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.679821 3735 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.780058 1447 raft_consensus.cc:1226] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
W20260808 01:10:48.784763 3767 consensus_meta.cc:287] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 1.198s user 0.017s sys 0.012s
I20260808 01:10:48.785326 3767 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.787062 2201 raft_consensus.cc:1226] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:48.783890 1445 raft_consensus.cc:1261] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:48.789271 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.789813 3767 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:48.790560 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.742722 3772 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:48.769161 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 65483us
I20260808 01:10:48.715925 1516 raft_consensus.cc:1226] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:48.665378 3784 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.777696 2204 raft_consensus.cc:1226] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
W20260808 01:10:48.778103 3791 consensus_meta.cc:287] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.932s user 0.016s sys 0.008s
I20260808 01:10:48.696928 1565 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:48.710470 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:48.811759 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 114935us
I20260808 01:10:48.805536 1489 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:48.800743 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:10:48.809926 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 153502us
I20260808 01:10:48.817978 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:48.819573 2204 raft_consensus.cc:1226] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:48.820783 2201 raft_consensus.cc:1226] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:48.823103 1565 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:10:48.828117 1490 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.019s sys 0.022s
W20260808 01:10:48.828500 1490 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.565s user 0.019s sys 0.022s
W20260808 01:10:48.828806 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 566042us
I20260808 01:10:48.810032 1446 raft_consensus.cc:2454] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:10:48.831483 1565 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:48.831794 1445 raft_consensus.cc:1261] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:48.832846 1565 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:48.835191 1565 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:48.830648 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.836110 1438 raft_consensus.cc:1226] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:48.836576 3789 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:48.839643 1442 raft_consensus.cc:1226] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: []
W20260808 01:10:48.809188 3756 consensus_meta.cc:287] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.965s user 0.011s sys 0.013s
I20260808 01:10:48.841176 1410 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:48.648686 3816 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.800805 3751 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:48.799760 3807 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:48.707720 3776 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:48.840188 3811 raft_consensus.cc:2726] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:48.840353 3804 raft_consensus.cc:2726] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:10:48.839574 3806 raft_consensus.cc:2726] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:48.839952 3742 raft_consensus.cc:2726] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:10:48.889267 3772 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:48.870494 3808 consensus_queue.cc:1046] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260808 01:10:48.892371 3764 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:48.893180 3808 raft_consensus.cc:3032] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:48.893712 3808 raft_consensus.cc:729] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:48.896219 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:48.897964 1410 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:48.829377 3814 raft_consensus.cc:2726] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:48.862057 1565 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:48.900715 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:48.828029 3792 consensus_queue.cc:1046] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260808 01:10:48.866415 1489 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:48.866515 3823 raft_consensus.cc:2726] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:48.867342 1603 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:48.867437 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:10:48.856804 3766 consensus_meta.cc:287] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.082s user 0.018s sys 0.003s
I20260808 01:10:48.905769 1603 raft_consensus.cc:595] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:48.906988 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:48.909696 1444 raft_consensus.cc:2379] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:48.847234 3697 consensus_queue.cc:1035] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
W20260808 01:10:48.911423 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 45078.9us
I20260808 01:10:48.882030 2205 raft_consensus.cc:1226] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 4. Current term is 5. Ops: []
W20260808 01:10:48.855203 3799 consensus_meta.cc:287] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.670s user 0.006s sys 0.009s
I20260808 01:10:48.879282 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.879652 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.880353 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.821460 3791 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:48.901460 3772 raft_consensus.cc:2726] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:10:48.887218 3785 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:10:48.871575 3779 consensus_meta.cc:287] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 1.362s user 0.039s sys 0.047s
I20260808 01:10:48.865535 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:48.744940 3819 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:48.925477 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:10:48.891620 3778 consensus_meta.cc:287] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.880s user 0.019s sys 0.004s
I20260808 01:10:48.827934 3818 consensus_queue.cc:1046] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260808 01:10:48.933099 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:48.904116 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:48.904286 1565 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:10:48.877737 3755 consensus_meta.cc:287] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.131s user 0.017s sys 0.014s
I20260808 01:10:48.839126 3821 consensus_queue.cc:1046] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260808 01:10:48.907090 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:48.872957 3789 raft_consensus.cc:2781] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:10:48.909584 1597 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:10:48.875630 1568 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:48.939932 1598 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:48.875020 3822 consensus_queue.cc:1046] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
W20260808 01:10:48.900226 3796 consensus_meta.cc:287] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.544s user 0.010s sys 0.016s
I20260808 01:10:48.913357 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:48.920328 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:48.894889 3808 consensus_queue.cc:260] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.915920 1489 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:48.881319 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:48.906723 3742 raft_consensus.cc:2726] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:48.920024 1442 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 559) took 1464 ms. Trace:
I20260808 01:10:48.902447 3784 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:48.863819 3756 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:48.904892 3735 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:48.911382 3804 raft_consensus.cc:1050] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:48.978821 1601 raft_consensus.cc:1226] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:48.943480 3823 consensus_queue.cc:1046] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260808 01:10:48.981570 3776 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:48.901777 1410 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:48.923187 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:48.877202 3803 consensus_queue.cc:1046] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260808 01:10:48.927666 1600 raft_consensus.cc:1226] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:48.928138 1448 raft_consensus.cc:1226] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:10:48.929288 3765 raft_consensus.cc:2781] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:48.931286 3819 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:48.912569 3766 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.936811 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:48.937551 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:48.937732 3818 raft_consensus.cc:3032] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Stepping down as leader of term 5
W20260808 01:10:48.940951 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 65432.5us
I20260808 01:10:48.919694 3792 consensus_queue.cc:1225] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer 95b661b03f1d460b9657e61d4b0247cd log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260808 01:10:48.915055 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 75649.5us
I20260808 01:10:48.958178 3820 raft_consensus.cc:3032] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:48.922125 3752 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:48.957963 3772 consensus_queue.cc:1046] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:10:48.904330 3826 consensus_queue.cc:1046] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260808 01:10:48.924892 3779 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.856704 3771 consensus_queue.cc:1035] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260808 01:10:48.969489 1442 rpcz_store.cc:276] 0808 01:10:47.451224 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:47.451351 (+ 127us) service_pool.cc:227] Handling call
0808 01:10:47.494429 (+ 43078us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:48.863180 (+1368751us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:48.915865 (+ 52685us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b540030c5d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:48.916036 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3080308608}
I20260808 01:10:48.962312 3796 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.976505 3791 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:48.976769 3756 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:10:48.983614 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 67816.4us
I20260808 01:10:48.942332 3748 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:10:48.984020 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 84119us
W20260808 01:10:48.986202 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 43933.2us
I20260808 01:10:48.941980 3789 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:48.947629 3814 raft_consensus.cc:2781] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:48.990655 1596 raft_consensus.cc:3037] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:48.993278 1568 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:48.938639 3801 consensus_queue.cc:1046] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.141s
I20260808 01:10:49.070091 2209 raft_consensus.cc:1226] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
W20260808 01:10:49.070271 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 77169.6us
I20260808 01:10:48.939910 3778 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:49.018585 3820 raft_consensus.cc:729] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:49.048244 3806 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:49.084847 2209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 615) took 1641 ms. Trace:
I20260808 01:10:49.085155 3806 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:48.988729 3808 raft_consensus.cc:3037] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:48.945001 3755 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:48.991285 3776 raft_consensus.cc:2726] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:10:48.989521 3803 raft_consensus.cc:3032] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:49.062642 1602 raft_consensus.cc:1226] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: []
I20260808 01:10:49.063930 3735 raft_consensus.cc:3032] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:10:49.085783 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:49.066576 1410 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:48.992551 3818 raft_consensus.cc:729] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:49.105958 1598 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:10:49.076061 3796 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:49.076891 3778 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:49.067039 3786 consensus_queue.cc:1046] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260808 01:10:49.078017 2201 raft_consensus.cc:1226] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:49.069815 3814 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:49.079427 1568 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:49.082862 1596 raft_consensus.cc:2454] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:10:49.052246 3752 consensus_queue.cc:1046] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260808 01:10:49.085317 2209 rpcz_store.cc:276] 0808 01:10:47.439243 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:47.439433 (+ 190us) service_pool.cc:227] Handling call
0808 01:10:47.444835 (+ 5402us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:49.070522 (+1625687us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:49.080046 (+ 9524us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b54007d05d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:49.080290 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3737274112}
W20260808 01:10:49.051091 3809 consensus_meta.cc:287] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.557s user 0.005s sys 0.011s
I20260808 01:10:49.057461 3822 consensus_queue.cc:1225] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer 95b661b03f1d460b9657e61d4b0247cd log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260808 01:10:49.089583 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:49.062668 3828 raft_consensus.cc:3032] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:48.982069 3804 consensus_queue.cc:237] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 3, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:49.098203 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:49.065062 3802 consensus_meta.cc:287] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.724s user 0.005s sys 0.015s
I20260808 01:10:49.069695 3789 raft_consensus.cc:3037] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:49.099802 3735 raft_consensus.cc:729] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:49.102254 3816 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:49.096678 3803 raft_consensus.cc:729] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:48.921257 3799 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:49.087903 3823 raft_consensus.cc:2781] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:10:49.076804 3791 consensus_queue.cc:1046] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260808 01:10:49.098939 3820 consensus_queue.cc:260] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:49.107218 1599 raft_consensus.cc:1226] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 1. Current term is 2. Ops: []
W20260808 01:10:49.193347 3795 consensus_meta.cc:287] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.255s user 0.012s sys 0.012s
I20260808 01:10:49.161579 3781 raft_consensus.cc:3032] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:49.266703 3781 raft_consensus.cc:729] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:49.267343 3809 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:49.269240 2207 raft_consensus.cc:2379] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:10:49.269366 3776 consensus_queue.cc:1046] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260808 01:10:49.270993 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 204502us
I20260808 01:10:49.304656 1439 raft_consensus.cc:1226] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
W20260808 01:10:49.126147 3780 consensus_meta.cc:287] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.803s user 0.012s sys 0.003s
I20260808 01:10:49.131417 3797 raft_consensus.cc:2781] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 01:10:49.311920 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 251637us
W20260808 01:10:49.307622 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 228269us
W20260808 01:10:49.193934 3769 consensus_meta.cc:287] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 1.447s user 0.016s sys 0.016s
I20260808 01:10:49.266306 3795 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:49.159409 3742 raft_consensus.cc:3032] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:49.313683 1568 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:49.325970 1568 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:49.225086 3765 raft_consensus.cc:686] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:49.316684 1442 raft_consensus.cc:1261] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:49.316919 1594 raft_consensus.cc:3032] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:49.316766 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:49.303627 3820 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:49.274479 3814 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:49.231112 3806 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:49.162963 3766 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:49.321313 3779 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:49.273202 3812 raft_consensus.cc:3032] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:49.274262 3823 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:49.155052 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:49.275514 3828 raft_consensus.cc:729] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:49.349819 1594 raft_consensus.cc:729] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:49.412863 1568 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:49.413265 1604 raft_consensus.cc:1226] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 7. Current term is 8. Ops: []
I20260808 01:10:49.356758 1439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55042 (request call id 2212) took 1123 ms. Trace:
I20260808 01:10:49.327718 2207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 928) took 1102 ms. Trace:
I20260808 01:10:49.414181 2207 rpcz_store.cc:276] 0808 01:10:48.214353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:48.214621 (+ 268us) service_pool.cc:227] Handling call
0808 01:10:49.316961 (+1102340us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54005de3d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:49.317100 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2399473280}
I20260808 01:10:49.357053 3823 raft_consensus.cc:3037] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:49.417295 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:49.306663 3755 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:49.104586 3826 consensus_queue.cc:1046] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.212s
I20260808 01:10:49.420224 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:49.164369 3792 consensus_queue.cc:1046] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.186s
W20260808 01:10:49.365664 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 59573.6us
I20260808 01:10:49.312182 3797 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:49.321732 3777 consensus_queue.cc:1046] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:49.309974 3751 raft_consensus.cc:2726] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:10:49.446298 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:49.423002 1439 rpcz_store.cc:276] 0808 01:10:48.233119 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:48.235941 (+ 2822us) service_pool.cc:227] Handling call
0808 01:10:48.236211 (+ 270us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:49.305366 (+1069155us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:49.356519 (+ 51153us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54002beb54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:49.356696 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2456679808}
I20260808 01:10:49.275406 3783 consensus_queue.cc:1046] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260808 01:10:49.351253 3752 raft_consensus.cc:3032] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:49.447887 3752 raft_consensus.cc:729] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
W20260808 01:10:49.447903 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 83024.8us
I20260808 01:10:49.448881 3752 consensus_queue.cc:260] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:49.450943 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:49.451223 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:49.326855 3742 raft_consensus.cc:729] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:49.412154 3830 raft_consensus.cc:3032] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:49.312363 3780 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:49.454586 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:49.464944 1517 raft_consensus.cc:3037] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:49.455328 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:49.456243 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:49.424510 3803 consensus_queue.cc:260] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:49.430159 3809 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:49.271189 3802 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:49.352856 3756 raft_consensus.cc:2726] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:49.429203 3806 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:49.360885 3791 raft_consensus.cc:2726] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:10:49.485884 2202 raft_consensus.cc:2379] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 3: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:10:49.126042 3818 consensus_queue.cc:260] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:49.454710 3722 consensus_meta.cc:287] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.722s user 0.012s sys 0.001s
I20260808 01:10:49.440989 3797 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:10:49.472559 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 59798.7us
I20260808 01:10:49.477319 1517 raft_consensus.cc:2454] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:10:49.440945 3755 raft_consensus.cc:2726] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:10:49.479398 3772 raft_consensus.cc:3032] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
W20260808 01:10:49.479620 3764 consensus_meta.cc:287] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.587s user 0.003s sys 0.011s
I20260808 01:10:49.501613 2208 raft_consensus.cc:1226] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:49.354173 3801 consensus_queue.cc:1046] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.113s
I20260808 01:10:49.323221 3811 consensus_queue.cc:1046] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.129s
I20260808 01:10:49.450363 3752 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:49.428944 3812 raft_consensus.cc:729] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:49.495043 3807 consensus_queue.cc:1046] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260808 01:10:49.521452 3780 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:49.516505 3803 raft_consensus.cc:3037] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:49.488276 1439 raft_consensus.cc:1226] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: []
I20260808 01:10:49.411928 3786 consensus_queue.cc:1035] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.105s
I20260808 01:10:49.500526 3778 raft_consensus.cc:3032] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:10:49.524228 3778 raft_consensus.cc:729] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:49.525235 3801 raft_consensus.cc:3032] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:49.452162 3829 consensus_queue.cc:1046] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.395s
I20260808 01:10:49.495227 3806 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:49.428812 3766 consensus_queue.cc:1046] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260808 01:10:49.496057 2206 raft_consensus.cc:1226] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:49.471088 2205 raft_consensus.cc:2379] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 3: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:10:49.497160 1594 consensus_queue.cc:260] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:49.424017 3781 consensus_queue.cc:260] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:49.481329 3802 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:49.324195 3769 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:49.359130 3735 consensus_queue.cc:260] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:49.501806 2202 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 925) took 1285 ms (client timeout 1661 ms). Trace:
I20260808 01:10:49.512122 1568 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:49.459187 3830 raft_consensus.cc:729] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:49.525714 3799 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:49.502171 3772 raft_consensus.cc:729] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
W20260808 01:10:49.561043 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 129345us
I20260808 01:10:49.733078 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:49.733812 1410 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:10:49.709442 1490 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.012s sys 0.012s
W20260808 01:10:49.734934 1490 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.749s user 0.012s sys 0.015s
I20260808 01:10:49.734251 3735 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:49.736161 1490 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:49.612278 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 514) took 1155 ms. Trace:
I20260808 01:10:49.618712 1517 raft_consensus.cc:1226] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:49.737035 2206 rpcz_store.cc:276] 0808 01:10:48.456585 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:48.756641 (+300056us) service_pool.cc:227] Handling call
0808 01:10:48.757062 (+ 421us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:49.557382 (+800320us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:49.611963 (+ 54581us) spinlock_profiling.cc:243] Waited 739 ms on lock 0x7b54007378d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:49.612200 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1698902272}
W20260808 01:10:49.737910 2202 rpcz_store.cc:269] 0808 01:10:48.213510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:48.213643 (+ 133us) service_pool.cc:227] Handling call
0808 01:10:49.498732 (+1285089us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400440f54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:49.498981 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2911626112}
I20260808 01:10:49.525880 3801 raft_consensus.cc:729] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:49.600581 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 515) took 1077 ms. Trace:
I20260808 01:10:49.739107 2208 rpcz_store.cc:276] 0808 01:10:48.500841 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:48.774599 (+273758us) service_pool.cc:227] Handling call
0808 01:10:49.498885 (+724286us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:49.508366 (+ 9481us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:49.578033 (+ 69667us) spinlock_profiling.cc:243] Waited 686 ms on lock 0x7b5400440f50. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:49.578264 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1577911040}
I20260808 01:10:49.584874 1594 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:49.562778 1521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58250 (request call id 1065) took 1120 ms. Trace:
W20260808 01:10:49.560871 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 327028us
I20260808 01:10:49.742453 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:49.744588 2202 raft_consensus.cc:2379] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 3: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
W20260808 01:10:49.748821 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 236817us
I20260808 01:10:49.749193 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:49.740419 1521 rpcz_store.cc:276] 0808 01:10:48.428524 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:48.429759 (+ 1235us) service_pool.cc:227] Handling call
0808 01:10:49.530409 (+1100650us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540023c8d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4af0 00007fe34ca47f2a 00007fe351984c6e 00007fe34b07b807 00007fe34b07b792 00007fe34b07b6c1 00007fe34b07b63c 00007fe34b07a8f2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:49.548355 (+ 17946us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2359264128}
I20260808 01:10:49.750046 1518 raft_consensus.cc:2379] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 2: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:10:49.763262 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:49.763978 1516 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:49.525777 3778 consensus_queue.cc:260] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:49.778357 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:49.774788 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:49.734506 1523 raft_consensus.cc:2454] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 1.
I20260808 01:10:49.770661 1490 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:10:49.577731 3831 log.cc:924] Time spent T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.304s user 0.001s sys 0.000s
I20260808 01:10:49.782958 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:49.576915 3826 consensus_queue.cc:1225] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer 95b661b03f1d460b9657e61d4b0247cd log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260808 01:10:49.783684 1522 raft_consensus.cc:1226] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:49.559057 3816 raft_consensus.cc:3032] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:10:49.785614 1568 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:49.494266 3765 consensus_queue.cc:237] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:49.810776 1519 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:49.790673 1490 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:49.714902 3787 raft_consensus.cc:3032] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:49.825981 1490 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:49.605422 3806 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:49.486617 3828 consensus_queue.cc:260] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:49.550359 3755 consensus_queue.cc:1046] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.258s
I20260808 01:10:49.781350 3781 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:49.783943 3833 raft_consensus.cc:2726] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:49.776086 3791 raft_consensus.cc:2781] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:49.807896 3792 raft_consensus.cc:2726] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:10:49.933377 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 147885us
W20260808 01:10:49.748611 3774 consensus_queue.cc:1159] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a4862d0938c84fc4bd840e4c17acae78" responder_term: 4 status { last_received { term: 4 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 4 index: 7 } }
I20260808 01:10:49.861840 3826 raft_consensus.cc:2726] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:49.829025 3756 raft_consensus.cc:2726] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:10:49.941793 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 229336us
I20260808 01:10:49.787554 3827 consensus_queue.cc:1046] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.142s
I20260808 01:10:49.826081 3795 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:49.813395 3816 raft_consensus.cc:729] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:49.825824 3787 raft_consensus.cc:729] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:49.828528 3807 raft_consensus.cc:2726] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:10:49.940912 1568 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:49.787931 3786 consensus_queue.cc:1046] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.146s
I20260808 01:10:49.927309 3722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:49260 (request call id 499) took 5509 ms. Trace:
I20260808 01:10:49.957919 2208 raft_consensus.cc:1226] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:49.959041 1522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 1997) took 1759 ms. Trace:
I20260808 01:10:49.959573 1522 rpcz_store.cc:276] 0808 01:10:48.190950 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:48.193015 (+ 2065us) service_pool.cc:227] Handling call
0808 01:10:48.193402 (+ 387us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:49.810522 (+1617120us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:49.949822 (+139300us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b54000a69d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:49.949959 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3656816512}
I20260808 01:10:49.808125 3812 consensus_queue.cc:260] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:49.968778 3812 raft_consensus.cc:3037] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:49.973644 1594 raft_consensus.cc:1261] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:49.974524 1594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 489) took 1101 ms. Trace:
I20260808 01:10:49.814124 3842 consensus_queue.cc:1046] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.161s
I20260808 01:10:49.974922 1594 rpcz_store.cc:276] 0808 01:10:48.872856 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:48.977115 (+104259us) service_pool.cc:227] Handling call
0808 01:10:48.977543 (+ 428us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:49.316806 (+339263us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b54002ff1a0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34c9cf730 00007fe34ca41300 00007fe34ca42371 00007fe34ca3eedf 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc
0808 01:10:49.485059 (+168253us) spinlock_profiling.cc:243] Waited 51.6 ms on lock 0x7b54002ff1a0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34c9d5951 00007fe34ca36321 00007fe34ca37b4e 00007fe34ca41c89 00007fe34ca42556 00007fe34ca3eedf 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612
0808 01:10:49.974184 (+489125us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:49.974441 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":798985856}
I20260808 01:10:49.939744 3791 raft_consensus.cc:686] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:49.740015 3801 consensus_queue.cc:260] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:50.000717 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:49.929764 3844 raft_consensus.cc:2726] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:50.005617 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260808 01:10:50.007721 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:50.007512 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:50.008481 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:50.008579 1440 raft_consensus.cc:2379] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
W20260808 01:10:49.989980 3819 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260808 01:10:50.015257 1442 raft_consensus.cc:1226] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:49.990952 3795 raft_consensus.cc:2726] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:50.017645 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:50.016907 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:50.021608 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:50.022140 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:49.909210 3830 consensus_queue.cc:260] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:50.037102 3772 consensus_queue.cc:260] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:49.984195 3833 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:50.020213 3791 consensus_queue.cc:237] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:50.060235 3833 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:50.008705 1438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 582) took 1071 ms. Trace:
W20260808 01:10:50.072674 3748 consensus_meta.cc:287] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.002s user 0.004s sys 0.011s
W20260808 01:10:50.075696 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 120558us
I20260808 01:10:50.075814 1516 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:50.069839 1519 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:10:50.031157 3797 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:50.084724 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 222593us
I20260808 01:10:50.098954 1524 raft_consensus.cc:3032] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:50.099435 1524 raft_consensus.cc:729] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:50.003340 3827 consensus_queue.cc:1046] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
W20260808 01:10:50.082973 3823 consensus_meta.cc:287] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.667s user 0.009s sys 0.004s
I20260808 01:10:50.014114 3787 consensus_queue.cc:260] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:50.121754 1490 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:50.087007 3720 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:50.083227 3748 raft_consensus.cc:2726] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:10:50.135020 1487 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.506s user 0.006s sys 0.013s
I20260808 01:10:50.071074 1438 rpcz_store.cc:276] 0808 01:10:48.932130 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:48.932375 (+ 245us) service_pool.cc:227] Handling call
0808 01:10:48.936230 (+ 3855us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:10:48.936924 (+ 694us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:48.937001 (+ 77us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:10:48.937784 (+ 783us) log.cc:842] Serialized 26 byte log entry
0808 01:10:49.069421 (+131637us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:10:49.070987 (+ 1566us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:49.084582 (+ 13595us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:10:49.990842 (+906260us) raft_consensus.cc:1671] finished
0808 01:10:49.992901 (+ 2059us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:50.003201 (+ 10300us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2970,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":17288,"thread_start_us":111731,"threads_started":1,"wal-append.queue_time_us":248187}
I20260808 01:10:50.005378 3845 consensus_queue.cc:1046] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.129s
I20260808 01:10:50.102432 3769 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:50.137003 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 507835us
I20260808 01:10:50.140142 3742 consensus_queue.cc:260] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:50.140514 3720 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:49.957897 3722 rpcz_store.cc:276] 0808 01:10:44.359919 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:44.360033 (+ 114us) service_pool.cc:227] Handling call
0808 01:10:44.451796 (+ 91763us) ts_tablet_manager.cc:928] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
0808 01:10:45.272935 (+821139us) spinlock_profiling.cc:243] Waited 1.45 ms on lock 0x7fe349ba2290. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349b3f641 00007fe349b15dde 00007fe349b13534 00007fe351930691 00007fe351936ef8 00007fe3519cc81e 00007fe3519fda8e 00007fe3519e35ad 00007fe3519e3559 00007fe3519e34e9
0808 01:10:47.584071 (+2311136us) tablet_copy_client.cc:361] Tablet copy session begun
0808 01:10:48.716785 (+1132714us) tablet_metadata.cc:641] Metadata flushed
0808 01:10:48.732733 (+ 15948us) tablet_copy_client.cc:490] Wrote new tablet metadata
0808 01:10:49.503137 (+770404us) tablet_copy_client.cc:411] Wrote new consensus metadata
0808 01:10:49.595518 (+ 92381us) tablet_replica.cc:205] Creating consensus instance
0808 01:10:49.778206 (+182688us) ts_tablet_manager.cc:983] Replica registered.
0808 01:10:49.868821 (+ 90615us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":244,"spinlock_wait_cycles":3326336,"tablet-copy.queue_time_us":90645,"thread_start_us":108549,"threads_started":2}
I20260808 01:10:50.149927 3787 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:50.139506 3839 raft_consensus.cc:3032] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:50.135911 3853 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:50.157894 1490 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:50.157882 1439 raft_consensus.cc:1226] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
W20260808 01:10:50.159657 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 349635us
I20260808 01:10:50.122378 3772 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:50.045117 3816 consensus_queue.cc:260] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:50.200714 2171 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:50.206923 2171 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:50.208593 1599 raft_consensus.cc:1226] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:50.209908 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:50.211719 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:50.212466 1601 raft_consensus.cc:1226] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:50.213515 1602 raft_consensus.cc:1226] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 1. Current term is 2. Ops: []
W20260808 01:10:50.220192 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 62386.7us
I20260808 01:10:50.243616 1490 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:50.247466 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:50.134047 3823 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:50.201779 3722 tablet_copy_client.cc:791] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 0 data blocks...
I20260808 01:10:50.201328 3706 tablet_bootstrap.cc:700] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd: Time spent opening tablet: real 0.204s user 0.001s sys 0.001s
I20260808 01:10:50.255151 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:50.527197 1599 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:50.217360 2171 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:50.253122 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:50.261511 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:10:50.528228 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 310912us
I20260808 01:10:50.530098 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:50.263077 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:50.531103 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:50.264283 3833 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:50.202257 3795 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:50.203579 3839 raft_consensus.cc:729] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:50.244122 3809 raft_consensus.cc:2781] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election won for term 2
W20260808 01:10:50.365195 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 97489.9us
W20260808 01:10:50.525475 3803 consensus_meta.cc:287] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 1.000s user 0.010s sys 0.007s
I20260808 01:10:50.368321 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:50.252719 3845 raft_consensus.cc:2781] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:50.426972 1446 raft_consensus.cc:1226] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:50.430971 3841 raft_consensus.cc:2781] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 01:10:50.461047 3860 log.cc:924] Time spent T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.186s user 0.000s sys 0.002s
W20260808 01:10:50.465744 3752 consensus_meta.cc:287] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.916s user 0.012s sys 0.003s
W20260808 01:10:50.414326 3831 log.cc:924] Time spent T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.159s user 0.000s sys 0.000s
I20260808 01:10:50.376736 3799 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:50.243323 3853 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:10:50.499910 3820 consensus_meta.cc:287] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 1.077s user 0.013s sys 0.001s
I20260808 01:10:50.500869 3720 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:10:50.502121 3808 consensus_meta.cc:287] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 1.415s user 0.008s sys 0.009s
I20260808 01:10:50.503659 3802 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:50.504196 3689 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:10:50.493201 3735 consensus_meta.cc:287] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.757s user 0.015s sys 0.001s
I20260808 01:10:50.214095 3784 raft_consensus.cc:2726] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:10:50.501991 3797 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:50.513162 3783 raft_consensus.cc:3032] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Stepping down as leader of term 2
W20260808 01:10:50.518770 3814 consensus_meta.cc:287] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.095s user 0.014s sys 0.003s
I20260808 01:10:50.524345 1603 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:50.529742 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
W20260808 01:10:50.538508 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 294972us
I20260808 01:10:50.542867 1604 raft_consensus.cc:2379] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 3: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:10:50.543828 2207 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:10:50.544008 2206 raft_consensus.cc:1226] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:50.612768 1490 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:50.542863 3722 tablet_copy_client.cc:655] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Starting download of 1 WAL segments...
I20260808 01:10:50.549167 1597 raft_consensus.cc:1226] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:50.366353 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:50.615391 1600 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:50.553119 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
W20260808 01:10:50.537882 3789 consensus_meta.cc:287] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 1.250s user 0.014s sys 0.007s
I20260808 01:10:50.560125 3859 raft_consensus.cc:2781] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:50.418908 3842 consensus_queue.cc:1046] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.114s
I20260808 01:10:50.572640 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:50.573160 3839 consensus_queue.cc:260] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:50.581195 1410 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.826s user 0.020s sys 0.081s
W20260808 01:10:50.632135 1410 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.877s user 0.020s sys 0.081s
I20260808 01:10:50.590286 1599 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:10:50.544625 3795 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:50.574277 3809 raft_consensus.cc:686] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:50.595773 3752 consensus_queue.cc:1046] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:10:50.592501 3841 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:50.601691 3820 raft_consensus.cc:2726] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:50.580447 3845 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:50.657196 3845 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:50.609421 2201 raft_consensus.cc:2379] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 2: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:10:50.605594 3799 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:50.594600 2205 raft_consensus.cc:1226] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:50.612655 2208 raft_consensus.cc:2379] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 2: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:10:50.614004 1490 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:50.610682 3783 raft_consensus.cc:729] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:50.618400 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:50.670532 1442 raft_consensus.cc:3037] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:50.620244 3833 raft_consensus.cc:2781] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:10:50.672129 3799 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:10:50.676188 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 62259us
I20260808 01:10:50.677275 1490 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:50.625207 1594 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:10:50.629319 3859 raft_consensus.cc:686] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
W20260808 01:10:50.632829 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 899056us
I20260808 01:10:50.694262 1410 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:50.643111 3840 raft_consensus.cc:3032] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:50.644137 1524 consensus_queue.cc:260] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:50.646745 3806 raft_consensus.cc:3032] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:50.705646 3806 raft_consensus.cc:729] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:50.706737 3818 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:50.646903 3778 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:50.619077 3814 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:50.621450 3823 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:10:50.600044 3812 consensus_meta.cc:287] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.631s user 0.012s sys 0.005s
I20260808 01:10:50.609386 3802 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:50.608783 3689 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:50.667819 3795 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:50.620730 3789 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:50.673246 3839 raft_consensus.cc:3037] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:50.685988 1490 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:50.689098 3853 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:50.687283 1525 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:10:50.653757 3841 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:50.702895 3735 raft_consensus.cc:2726] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:10:50.669935 3820 raft_consensus.cc:2726] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:10:50.717406 1603 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:10:50.747014 2202 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:50.697150 3844 raft_consensus.cc:2726] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:50.675734 3833 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:50.733980 1600 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:10:50.731775 1410 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:50.716336 3840 raft_consensus.cc:729] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:50.739877 1596 raft_consensus.cc:1226] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:50.741168 3769 raft_consensus.cc:2726] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:50.692842 3859 consensus_queue.cc:237] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:50.743103 3838 raft_consensus.cc:3032] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:50.639209 3809 consensus_queue.cc:237] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:50.758072 1410 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:10:50.739575 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 57202.7us
I20260808 01:10:50.769074 1490 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:50.707131 3806 consensus_queue.cc:260] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:50.773058 1410 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:50.775045 1410 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:50.782991 1490 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:50.758740 3833 raft_consensus.cc:3037] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:50.787230 3836 raft_consensus.cc:2726] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:50.801182 1490 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:50.802364 2202 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
W20260808 01:10:50.791924 3848 log.cc:924] Time spent T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.676s user 0.005s sys 0.011s
I20260808 01:10:50.793025 3789 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:50.793514 3806 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:50.793627 3689 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:50.794373 3807 raft_consensus.cc:2781] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election won for term 2
W20260808 01:10:50.794852 3856 log.cc:924] Time spent T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.234s user 0.000s sys 0.002s
I20260808 01:10:50.803514 1442 raft_consensus.cc:2454] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:50.828802 1568 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:50.830060 3807 raft_consensus.cc:686] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:50.832448 3735 raft_consensus.cc:2726] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:50.805039 3751 raft_consensus.cc:3032] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:50.780027 3769 raft_consensus.cc:2726] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:10:50.815752 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:50.820353 3797 consensus_queue.cc:1046] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260808 01:10:50.809813 3844 raft_consensus.cc:2726] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:50.784382 3838 raft_consensus.cc:729] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
W20260808 01:10:49.976651 3829 consensus_queue.cc:1159] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" responder_term: 2 status { last_received { term: 1 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:10:50.784809 3803 raft_consensus.cc:2781] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:10:50.820016 2204 raft_consensus.cc:2454] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:50.838910 3751 raft_consensus.cc:729] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:50.867476 3840 consensus_queue.cc:260] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:50.881965 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:50.882793 1446 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:50.882452 3751 consensus_queue.cc:260] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:50.873158 3836 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:50.853119 1448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 593) took 1320 ms. Trace:
I20260808 01:10:50.853163 3769 raft_consensus.cc:2726] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:10:50.850513 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 49429.5us
W20260808 01:10:50.863029 1525 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36560 (request call id 508) took 1428 ms (client timeout 1623 ms). Trace:
W20260808 01:10:50.894398 1525 rpcz_store.cc:269] 0808 01:10:49.434843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:49.435021 (+ 178us) service_pool.cc:227] Handling call
0808 01:10:50.862774 (+1427753us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540035d9d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:50.862913 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2592186624}
I20260808 01:10:50.894697 1490 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:50.896237 1490 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:50.884537 3751 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:50.890393 1448 rpcz_store.cc:276] 0808 01:10:49.532577 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:49.533010 (+ 433us) service_pool.cc:227] Handling call
0808 01:10:49.576808 (+ 43798us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:10:49.864489 (+287681us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:49.864536 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:10:49.905885 (+ 41349us) log.cc:842] Serialized 26 byte log entry
0808 01:10:50.094601 (+188716us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:10:50.094655 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:50.094754 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:10:50.852800 (+758046us) raft_consensus.cc:1671] finished
0808 01:10:50.852842 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:50.853046 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":137740,"raft.queue_time_us":185608,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":10631,"thread_start_us":266577,"threads_started":2,"wal-append.queue_time_us":280859}
I20260808 01:10:50.846832 3795 raft_consensus.cc:2781] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:50.866137 3828 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:50.851729 3797 raft_consensus.cc:3032] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:50.898150 3844 raft_consensus.cc:2726] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:50.884155 3803 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:50.932663 1490 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:50.816987 3862 consensus_queue.cc:1046] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.268s
I20260808 01:10:50.936609 1490 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:50.909569 3796 raft_consensus.cc:2726] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:10:50.913136 3827 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.777s user 0.026s sys 0.093s
I20260808 01:10:50.940064 1446 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:10:50.949319 1568 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:50.922560 3803 raft_consensus.cc:3037] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:50.919378 3814 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:50.963976 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:50.872905 3807 consensus_queue.cc:237] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:50.916764 3795 raft_consensus.cc:686] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:50.914381 3797 raft_consensus.cc:729] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:50.974067 1525 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
W20260808 01:10:50.960309 3827 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.815s user 0.026s sys 0.094s
W20260808 01:10:50.966058 3787 consensus_meta.cc:287] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.771s user 0.013s sys 0.006s
I20260808 01:10:50.948406 3769 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:50.945806 3764 raft_consensus.cc:2726] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:10:50.967195 3781 consensus_meta.cc:287] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 1.042s user 0.017s sys 0.003s
I20260808 01:10:51.000842 1524 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:51.032467 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:50.970604 3802 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:50.975574 3823 raft_consensus.cc:2781] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election won for term 4
W20260808 01:10:51.002169 3772 consensus_meta.cc:287] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.798s user 0.008s sys 0.013s
I20260808 01:10:50.900470 3836 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:51.016083 1523 raft_consensus.cc:1226] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:51.064687 3797 consensus_queue.cc:260] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:51.076865 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:51.077862 1525 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:10:51.038100 3774 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:51.050423 3787 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:51.103163 1517 raft_consensus.cc:1261] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:51.053443 3781 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:51.094447 3797 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:51.095536 3755 raft_consensus.cc:3032] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:10:51.063189 3834 consensus_queue.cc:1046] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.222s
I20260808 01:10:51.117014 3789 consensus_queue.cc:1046] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.068s
I20260808 01:10:51.099076 3774 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:51.156139 2204 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 632) took 2242 ms (client timeout 1598 ms). Trace:
I20260808 01:10:51.178504 1517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36560 (request call id 517) took 1435 ms. Trace:
I20260808 01:10:51.260878 1517 rpcz_store.cc:276] 0808 01:10:49.743261 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:49.743699 (+ 438us) service_pool.cc:227] Handling call
0808 01:10:49.744104 (+ 405us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:51.102785 (+1358681us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b54003e7220. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34c9cf730 00007fe34ca41300 00007fe34ca42371 00007fe34ca3eedf 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc
0808 01:10:51.115111 (+ 12326us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:51.178146 (+ 63035us) spinlock_profiling.cc:243] Waited 988 ms on lock 0x7b54000e9f54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:51.178422 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2614435072}
W20260808 01:10:51.255470 1487 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.052s user 0.020s sys 0.078s
W20260808 01:10:51.263897 1487 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.060s user 0.021s sys 0.078s
I20260808 01:10:51.138453 1524 raft_consensus.cc:1261] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:51.265409 1524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36560 (request call id 516) took 1819 ms. Trace:
I20260808 01:10:50.987497 3838 consensus_queue.cc:260] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:51.266048 1524 rpcz_store.cc:276] 0808 01:10:49.446343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:49.467953 (+ 21610us) service_pool.cc:227] Handling call
0808 01:10:49.468238 (+ 285us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:50.098840 (+630602us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b54003ec220. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34c9cf730 00007fe34ca41300 00007fe34ca42371 00007fe34ca3eedf 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc
0808 01:10:50.643697 (+544857us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54003ec220. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34c9d5951 00007fe34ca36321 00007fe34ca37b4e 00007fe34ca41c89 00007fe34ca42556 00007fe34ca3eedf 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612
0808 01:10:51.000527 (+356830us) spinlock_profiling.cc:243] Waited 27.2 ms on lock 0x7b58000af7b0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34c9baf63 00007fe34ca1eaba 00007fe34ca363af 00007fe34ca37b4e 00007fe34ca41c89 00007fe34ca42556 00007fe34ca3eedf 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684
0808 01:10:51.265028 (+264501us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:51.265340 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1133205632}
I20260808 01:10:51.653712 1523 raft_consensus.cc:1226] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
W20260808 01:10:51.654893 1489 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.288s user 0.001s sys 0.009s
W20260808 01:10:51.655637 1489 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.289s user 0.001s sys 0.009s
W20260808 01:10:51.656108 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 1.28948e+06us
I20260808 01:10:51.658069 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:51.660765 1523 raft_consensus.cc:3037] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:51.658736 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:51.661057 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:51.662143 1518 raft_consensus.cc:3037] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:51.664144 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:51.665195 1525 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
W20260808 01:10:51.660602 1565 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.557s user 0.037s sys 0.058s
W20260808 01:10:51.665782 1565 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.562s user 0.038s sys 0.058s
I20260808 01:10:51.666390 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:51.441304 1517 raft_consensus.cc:1226] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
W20260808 01:10:51.454950 2172 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.076s user 0.017s sys 0.050s
W20260808 01:10:51.679035 2172 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.300s user 0.017s sys 0.053s
W20260808 01:10:51.259076 2204 rpcz_store.cc:269] 0808 01:10:48.907676 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:48.908007 (+ 331us) service_pool.cc:227] Handling call
0808 01:10:51.149679 (+2241672us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b54006b03d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:51.149890 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3731706752}
I20260808 01:10:51.683008 3735 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:51.684270 3735 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:51.738334 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 52587.4us
W20260808 01:10:51.237859 3778 consensus_meta.cc:287] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.522s user 0.009s sys 0.004s
W20260808 01:10:51.691188 3771 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.540s user 0.009s sys 0.031s
I20260808 01:10:51.689495 3735 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:51.702178 3774 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:51.822871 3863 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:51.823711 3772 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:51.187278 3766 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:51.828773 3865 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:51.187397 3822 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:51.194128 3830 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:51.829512 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:51.265806 3836 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:51.451633 3837 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
W20260808 01:10:51.499794 3751 consensus_meta.cc:287] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.602s user 0.010s sys 0.005s
I20260808 01:10:51.853528 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:51.180577 3823 raft_consensus.cc:686] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:51.856040 3823 consensus_queue.cc:237] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:51.452036 3857 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:51.846154 1524 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:51.170693 3861 raft_consensus.cc:3032] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:51.836071 1518 raft_consensus.cc:2454] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
W20260808 01:10:51.244869 3839 consensus_meta.cc:287] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.501s user 0.005s sys 0.011s
W20260808 01:10:51.590360 3841 consensus_meta.cc:287] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.832s user 0.011s sys 0.006s
W20260808 01:10:51.577337 3791 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.893s user 0.006s sys 0.022s
W20260808 01:10:51.242794 3818 consensus_meta.cc:287] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.535s user 0.013s sys 0.002s
W20260808 01:10:51.587189 3803 consensus_meta.cc:287] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.592s user 0.013s sys 0.001s
W20260808 01:10:51.613272 3845 consensus_meta.cc:287] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.940s user 0.006s sys 0.012s
I20260808 01:10:51.847193 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:51.853860 1521 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:51.833206 3756 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:10:51.844213 3771 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.690s user 0.014s sys 0.032s
I20260808 01:10:51.190987 3850 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:51.175405 3801 raft_consensus.cc:3037] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:10:51.861228 1444 consensus_meta.cc:287] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.826s user 0.011s sys 0.004s
I20260808 01:10:51.203821 3854 raft_consensus.cc:3032] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:10:51.193604 3816 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:51.183444 3755 raft_consensus.cc:729] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:51.170298 3783 consensus_queue.cc:260] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:51.522722 3799 raft_consensus.cc:3032] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 LEADER]: Stepping down as leader of term 7
W20260808 01:10:51.863035 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 137425us
I20260808 01:10:51.889183 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:51.260053 3802 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:10:51.870488 1567 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.751s user 0.017s sys 0.032s
I20260808 01:10:51.872867 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:10:51.455183 3765 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.076s user 0.019s sys 0.066s
I20260808 01:10:51.176134 3742 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:51.894568 3789 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:51.894284 3756 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:51.895644 3789 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:51.897493 3756 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:51.898066 1408 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.376s user 0.015s sys 0.022s
I20260808 01:10:51.898703 3789 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:51.899235 3854 raft_consensus.cc:729] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:51.899642 3799 raft_consensus.cc:729] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:51.900074 3748 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:51.900975 3838 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:10:51.902894 1567 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.784s user 0.017s sys 0.032s
W20260808 01:10:51.879680 1568 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.730s (SENT) [suppressed 10 similar messages]
W20260808 01:10:51.910725 1408 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.389s user 0.016s sys 0.022s
I20260808 01:10:51.159519 3781 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:51.915458 3781 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:51.915959 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:10:51.918059 3791 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.223s user 0.012s sys 0.024s
I20260808 01:10:51.918610 3735 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:51.921254 3836 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:51.923481 3772 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:51.931990 3748 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:51.933990 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:51.936118 3840 raft_consensus.cc:3037] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:51.936750 1518 raft_consensus.cc:2379] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 3: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:10:51.936937 3799 consensus_queue.cc:260] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:51.901068 3850 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:51.938593 3863 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:51.934893 3841 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:51.944774 3861 raft_consensus.cc:729] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
W20260808 01:10:51.941665 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 100014us
I20260808 01:10:51.906736 1444 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:51.949009 1517 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:51.957626 3857 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:51.953537 3735 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:51.940117 3755 consensus_queue.cc:260] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:51.954464 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:51.872664 3822 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:51.966174 3845 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:51.968802 3822 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:51.970234 1519 raft_consensus.cc:1226] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:51.972772 1598 raft_consensus.cc:1226] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:51.973404 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:51.974502 1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2387) took 1765 ms. Trace:
I20260808 01:10:51.974916 1598 rpcz_store.cc:276] 0808 01:10:50.208821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:50.213737 (+ 4916us) service_pool.cc:227] Handling call
0808 01:10:50.214041 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:51.973330 (+1759289us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:51.974093 (+ 763us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b5400304b54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:51.974339 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4043391872}
W20260808 01:10:51.962622 3804 consensus_queue.cc:1159] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a4862d0938c84fc4bd840e4c17acae78" responder_term: 2 status { last_received { term: 2 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 2 index: 5 } }
I20260808 01:10:51.977591 3735 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:51.978348 1489 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:51.979214 3854 consensus_queue.cc:260] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:51.981570 1520 raft_consensus.cc:1226] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:10:51.982093 3854 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:51.969027 3845 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:51.963932 3803 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:51.984733 3845 raft_consensus.cc:2726] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:10:51.984900 3841 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:10:51.985699 3833 consensus_meta.cc:287] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 1.166s user 0.006s sys 0.013s
I20260808 01:10:51.986203 3833 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:51.987460 1442 raft_consensus.cc:1226] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:51.988277 1522 raft_consensus.cc:1226] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:51.989336 1442 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1922) took 1150 ms. Trace:
I20260808 01:10:51.964844 3766 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:51.951887 3865 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:51.992249 3766 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:51.992726 3865 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:51.989800 1522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1779) took 1026 ms. Trace:
I20260808 01:10:51.986546 3803 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:51.987721 1524 raft_consensus.cc:1226] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:52.011281 1525 raft_consensus.cc:2454] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
W20260808 01:10:51.952493 3828 consensus_meta.cc:287] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.927s user 0.012s sys 0.008s
I20260808 01:10:51.952939 3789 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:51.989830 1442 rpcz_store.cc:276] 0808 01:10:50.838590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:50.839519 (+ 929us) service_pool.cc:227] Handling call
0808 01:10:50.839996 (+ 477us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:51.988111 (+1148115us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:51.988972 (+ 861us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54001bf3d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:51.989239 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2637717760}
W20260808 01:10:51.966183 1444 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55040 (request call id 1908) took 1966 ms (client timeout 1730 ms). Trace:
W20260808 01:10:52.021585 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 43532.4us
I20260808 01:10:51.946362 3774 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:52.024091 3799 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:10:52.024817 3863 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:51.993185 3748 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:10:52.020848 1444 rpcz_store.cc:269] 0808 01:10:49.999201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:50.000378 (+ 1177us) service_pool.cc:227] Handling call
0808 01:10:51.965739 (+1965361us) spinlock_profiling.cc:243] Waited 992 ms on lock 0x7b540002e9d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:51.965961 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2281699712}
I20260808 01:10:52.011714 1522 rpcz_store.cc:276] 0808 01:10:50.963881 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:50.964048 (+ 167us) service_pool.cc:227] Handling call
0808 01:10:50.964362 (+ 314us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:51.988797 (+1024435us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:51.989510 (+ 713us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54003101d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:51.989740 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2353722240}
W20260808 01:10:51.971773 3797 consensus_meta.cc:287] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.787s user 0.006s sys 0.006s
I20260808 01:10:52.042038 3774 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.051456 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:10:52.020038 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 1.50813e+06us
I20260808 01:10:51.151036 3787 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:51.167588 3795 consensus_queue.cc:237] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:52.072809 3766 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:52.073673 3766 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.074070 3802 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.077241 1523 raft_consensus.cc:2454] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:52.077750 3802 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:10:52.067392 1488 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.665s user 0.017s sys 0.026s
W20260808 01:10:52.105636 1488 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.884s user 0.018s sys 0.026s
W20260808 01:10:52.084184 2170 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.482s user 0.023s sys 0.028s
W20260808 01:10:52.106261 1411 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.907s user 0.003s sys 0.034s
I20260808 01:10:52.073172 3833 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:10:52.089264 3765 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.633s user 0.028s sys 0.079s
W20260808 01:10:52.116855 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 274213us
W20260808 01:10:52.116583 2170 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.740s user 0.024s sys 0.028s
I20260808 01:10:52.065038 3748 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:52.089695 3766 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:52.115142 2171 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.365s user 0.049s sys 0.060s
W20260808 01:10:52.123886 1411 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.127s user 0.003s sys 0.034s
W20260808 01:10:52.095578 3859 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.716s user 0.025s sys 0.019s
W20260808 01:10:52.103780 3807 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.014s sys 0.012s
I20260808 01:10:52.133193 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:52.137789 3769 consensus_meta.cc:287] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 1.127s user 0.011s sys 0.011s
I20260808 01:10:52.138106 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:52.139621 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:52.141074 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:52.141987 2208 raft_consensus.cc:2454] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:52.142763 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.138365 3769 raft_consensus.cc:2726] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:10:52.117851 3755 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:10:52.115017 3809 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.834s user 0.019s sys 0.014s
I20260808 01:10:52.156627 3774 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:52.155133 2171 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.616s user 0.050s sys 0.060s
I20260808 01:10:52.130723 3797 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:52.164739 3797 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:52.166047 1516 raft_consensus.cc:1226] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
W20260808 01:10:52.166836 3877 log.cc:924] Time spent T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.093s user 0.002s sys 0.000s
I20260808 01:10:52.167124 3797 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:52.167889 1516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1780) took 1196 ms. Trace:
I20260808 01:10:52.168009 3748 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:52.168318 1516 rpcz_store.cc:276] 0808 01:10:50.971924 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:50.972103 (+ 179us) service_pool.cc:227] Handling call
0808 01:10:50.972491 (+ 388us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:52.166525 (+1194034us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:52.167557 (+ 1032us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54001a1b54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:52.167787 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2731321216}
I20260808 01:10:52.170296 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:52.170722 3748 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:52.171103 1522 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:52.089949 3789 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:52.173379 1567 connection.cc:232] RPC call timeout handler was delayed by -1.8499671430326998s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1648s for a response
I20260808 01:10:52.173669 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:52.173743 3872 consensus_queue.cc:1046] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:52.174966 1521 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
W20260808 01:10:52.157730 3859 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.981s user 0.026s sys 0.020s
W20260808 01:10:52.160392 3807 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.959s user 0.015s sys 0.013s
I20260808 01:10:52.067947 3836 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.188103 3783 raft_consensus.cc:3037] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:52.090459 3861 consensus_queue.cc:260] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:52.194003 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:52.181656 3809 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.107s user 0.020s sys 0.015s
I20260808 01:10:52.224999 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:52.226439 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 44799.2us
I20260808 01:10:52.227270 1410 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:52.228521 3865 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:52.229212 3865 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.231438 3865 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.232358 3865 consensus_queue.cc:1046] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:10:52.233204 3836 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.235014 3765 raft_consensus.cc:3032] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:52.235459 3765 raft_consensus.cc:729] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:52.236596 3765 consensus_queue.cc:260] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:52.237746 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.238266 3765 raft_consensus.cc:3037] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:52.238654 1442 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:10:52.240031 3774 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:52.240720 3774 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:52.241216 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 1.70248e+06us
I20260808 01:10:52.241956 3801 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:52.242619 3801 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.243160 3774 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.244357 1410 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:52.244911 3801 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.245431 3801 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:52.245807 2171 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:52.246124 3801 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:52.197857 3821 consensus_queue.cc:1046] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:10:52.199977 3859 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:52.201776 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:52.201893 2204 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:10:52.203245 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:52.204852 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.205273 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:52.205837 3789 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.208420 3778 consensus_queue.cc:1046] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:10:52.270349 3778 raft_consensus.cc:3032] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:52.270732 3778 raft_consensus.cc:729] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:52.270958 1444 raft_consensus.cc:1226] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:52.271454 3833 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:52.271991 3836 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:52.271759 3778 consensus_queue.cc:260] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.272069 3833 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.273234 3778 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:52.273383 1523 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:52.212070 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.212582 3850 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.213040 3857 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:52.265664 3859 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.278931 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.213840 3787 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.279735 3833 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.216081 3880 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:52.279783 1523 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:10:52.280879 3859 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:52.280710 3880 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:52.283223 3880 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:52.221756 3861 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:52.246850 3802 raft_consensus.cc:2726] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:10:52.247648 3803 raft_consensus.cc:2781] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:52.289387 3803 raft_consensus.cc:686] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:52.248297 3884 raft_consensus.cc:2781] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:52.250125 3748 raft_consensus.cc:3032] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:52.250423 3852 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:52.250556 3816 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:52.252149 1440 raft_consensus.cc:1226] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:52.252278 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.256366 3772 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:52.258344 1443 raft_consensus.cc:2454] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:52.258791 1517 raft_consensus.cc:2454] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:10:52.318523 1440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1925) took 1430 ms. Trace:
I20260808 01:10:52.260686 1522 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:10:52.195785 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:10:52.261636 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 78381.5us
I20260808 01:10:52.264885 1519 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:10:52.265372 3821 raft_consensus.cc:3032] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:52.265525 3828 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:52.265445 1520 raft_consensus.cc:2379] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 3: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:10:52.265419 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.266706 3769 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:52.266979 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.275146 1516 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:10:52.272588 3836 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.321647 1524 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:10:52.286209 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.287029 3801 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:52.327652 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:52.328440 1442 raft_consensus.cc:2362] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2. Re-sending same reply.
I20260808 01:10:52.293591 3884 raft_consensus.cc:686] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:52.263147 3799 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:52.333118 3799 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.334122 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.335035 1442 raft_consensus.cc:2379] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 2: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:52.335187 3799 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.293918 3748 raft_consensus.cc:729] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:52.296284 3852 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.297711 3806 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:52.300486 3850 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:52.301851 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.302625 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.304382 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.309819 1525 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:52.314287 3706 tablet_bootstrap.cc:492] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:10:52.342826 1445 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:52.320107 3821 raft_consensus.cc:729] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:52.345964 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:52.346741 3821 consensus_queue.cc:260] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:52.324137 1448 raft_consensus.cc:2454] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:52.346596 1442 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:52.344326 1518 raft_consensus.cc:2454] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
W20260808 01:10:52.324657 1443 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55040 (request call id 1907) took 2325 ms (client timeout 1852 ms). Trace:
I20260808 01:10:52.325315 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.325497 3828 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.326795 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.327419 3855 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:52.352253 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.288555 1439 raft_consensus.cc:1226] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:10:52.327971 3769 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.290705 3803 consensus_queue.cc:237] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:52.332574 3836 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.336452 3884 consensus_queue.cc:237] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:52.363574 1594 raft_consensus.cc:1226] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
I20260808 01:10:52.363638 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.352174 3806 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.365063 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.366143 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.341583 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:52.337541 1521 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:52.368714 3828 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.369410 3836 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:52.369994 3836 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:52.352038 3850 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.318895 1440 rpcz_store.cc:276] 0808 01:10:50.887778 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:50.887962 (+ 184us) service_pool.cc:227] Handling call
0808 01:10:50.888284 (+ 322us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:52.315931 (+1427647us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:52.316655 (+ 724us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b540030bbd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:52.318458 (+ 1803us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3135332736}
I20260808 01:10:52.342034 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:52.374023 3850 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.345038 3742 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:52.374488 1440 raft_consensus.cc:1226] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:52.358518 3855 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:52.374806 3742 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.362548 1519 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 8.
I20260808 01:10:52.336771 3748 consensus_queue.cc:260] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:52.376813 3748 raft_consensus.cc:3037] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:52.377069 3742 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.345108 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.345583 1444 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:52.380270 3836 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:10:52.324352 1567 connection.cc:232] RPC call timeout handler was delayed by -1.7278403940144926s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21480000000000002s for a response
I20260808 01:10:52.355741 1598 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:52.381424 3706 tablet_bootstrap.cc:492] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:10:52.353180 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:52.353359 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:52.381889 1521 raft_consensus.cc:2454] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:52.382297 3706 ts_tablet_manager.cc:1397] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 3.641s user 0.047s sys 0.048s
I20260808 01:10:52.351799 1596 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:10:52.347687 3852 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:52.361378 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 115853us
W20260808 01:10:52.348812 1443 rpcz_store.cc:269] 0808 01:10:49.998853 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:50.000298 (+ 1445us) service_pool.cc:227] Handling call
0808 01:10:52.324411 (+2324113us) spinlock_profiling.cc:243] Waited 2.19 s on lock 0x7b54002bf554. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:52.324549 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5040257408}
I20260808 01:10:52.367108 3769 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.358565 3821 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:10:52.349006 1438 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.290s user 0.066s sys 0.170s
I20260808 01:10:52.350725 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:52.387857 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:52.389551 1448 raft_consensus.cc:2379] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 2: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:52.390131 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.391585 1439 raft_consensus.cc:2379] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 3: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:52.392590 1440 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:10:52.396155 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:10:52.396631 1567 connection.cc:232] RPC call timeout handler was delayed by -1.5131257330067456s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17300000000000001s for a response
W20260808 01:10:52.397020 1567 connection.cc:232] RPC call timeout handler was delayed by -1.466667660017265s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21440000000000003s for a response
I20260808 01:10:52.397595 3783 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:52.398213 3783 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.398707 3865 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
W20260808 01:10:52.399971 1567 connection.cc:232] RPC call timeout handler was delayed by -1.4033257330011111s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18520000000000003s for a response
W20260808 01:10:52.406174 1567 connection.cc:232] RPC call timeout handler was delayed by -1.2867555990233086s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16190000000000002s for a response
W20260808 01:10:52.406653 1567 connection.cc:232] RPC call timeout handler was delayed by -1.2776653140317649s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18400000000000002s for a response
I20260808 01:10:52.356598 3816 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:52.408237 1567 connection.cc:232] RPC call timeout handler was delayed by -1.2247855179884937s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2102s for a response
I20260808 01:10:52.408075 3852 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:52.409294 2171 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:10:52.410301 1567 connection.cc:232] RPC call timeout handler was delayed by -1.214049590023933s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18280000000000002s for a response
W20260808 01:10:52.399901 1438 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.534s user 0.066s sys 0.172s
I20260808 01:10:52.415320 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.416363 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.417213 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.419018 1440 raft_consensus.cc:1226] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:52.424052 1447 raft_consensus.cc:2454] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
W20260808 01:10:52.425449 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 252109us
I20260808 01:10:52.427402 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.429121 1596 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:52.427975 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.433876 1446 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:10:52.437515 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:52.437613 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:52.438640 1448 raft_consensus.cc:2362] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3. Re-sending same reply.
I20260808 01:10:52.439803 1489 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:52.441970 1596 raft_consensus.cc:2379] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 4: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:10:52.424638 3769 consensus_queue.cc:1046] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260808 01:10:52.441965 1490 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:52.430599 3778 raft_consensus.cc:2726] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:10:52.425686 3865 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.461449 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:52.453191 3769 raft_consensus.cc:3032] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:10:52.440780 3852 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.473385 1408 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:52.461364 3828 raft_consensus.cc:2781] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:10:52.471813 3765 raft_consensus.cc:2781] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:52.469853 3783 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:52.477646 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 85353.6us
I20260808 01:10:52.466341 3769 raft_consensus.cc:729] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:52.459533 3850 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:52.481344 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:52.482139 3818 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260808 01:10:52.482911 1568 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:52.485054 3855 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:10:52.485913 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 80031.1us
I20260808 01:10:52.492127 1596 raft_consensus.cc:2454] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:52.474457 1438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1923) took 1632 ms. Trace:
I20260808 01:10:52.477069 3828 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:52.478492 3765 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:52.497222 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:52.490724 3803 raft_consensus.cc:2781] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:52.499235 1408 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:52.501237 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.498082 1568 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:52.502290 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.502374 1410 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:52.508325 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.509526 1517 raft_consensus.cc:2454] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:52.510200 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.500762 3765 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:52.512060 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.497553 3828 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:52.512923 1522 raft_consensus.cc:2454] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:52.513396 3799 raft_consensus.cc:2781] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:10:52.513512 3855 raft_consensus.cc:2781] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:10:52.513881 3799 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:52.514381 3799 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:52.500061 3838 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:52.514729 3816 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.508569 1490 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:52.515408 3783 raft_consensus.cc:2781] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:10:52.498574 3861 raft_consensus.cc:2781] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:52.515360 3838 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.515986 3783 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:52.516003 3840 raft_consensus.cc:2726] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:52.516574 3783 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:52.517061 1521 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:52.517925 3838 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.518968 1490 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:52.515882 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:52.520304 1525 raft_consensus.cc:2379] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 2: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:10:52.521001 1600 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:52.514330 3855 raft_consensus.cc:686] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:52.524186 3861 raft_consensus.cc:686] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:52.525315 3806 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.525058 3855 consensus_queue.cc:237] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.526458 3840 raft_consensus.cc:2726] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:10:52.525200 3861 consensus_queue.cc:237] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:52.497095 3850 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:52.530694 1490 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:52.532022 3838 raft_consensus.cc:2781] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:52.532495 3838 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:52.532950 3838 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:52.537838 1490 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:52.541687 1568 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:52.543423 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:52.543397 1442 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:52.545190 1445 raft_consensus.cc:1226] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:52.501077 3803 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:52.550813 3806 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:10:52.551394 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:52.551548 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.552314 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:52.550896 1342 catalog_manager.cc:5495] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 3 to 4, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 4 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:52.553949 2208 raft_consensus.cc:2454] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:52.556002 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:52.557632 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.559149 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:52.560642 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 69644.3us
I20260808 01:10:52.562148 1521 raft_consensus.cc:2379] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 4: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:10:52.559365 1342 catalog_manager.cc:5495] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 3 to 4, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 4 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } }
I20260808 01:10:52.568372 1408 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:52.555747 3806 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:52.568938 2202 raft_consensus.cc:2362] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3. Re-sending same reply.
I20260808 01:10:52.569114 2209 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:52.555613 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:52.572477 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:52.496341 1438 rpcz_store.cc:276] 0808 01:10:50.842009 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:50.842954 (+ 945us) service_pool.cc:227] Handling call
0808 01:10:50.843409 (+ 455us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:10:50.864903 (+ 21494us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:50.864964 (+ 61us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:10:50.865808 (+ 844us) log.cc:842] Serialized 26 byte log entry
0808 01:10:52.439044 (+1573236us) raft_consensus.cc:1631] Marking committed up to 3
0808 01:10:52.448963 (+ 9919us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:52.459459 (+ 10496us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:10:52.460658 (+ 1199us) raft_consensus.cc:1671] finished
0808 01:10:52.462054 (+ 1396us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:52.472890 (+ 10836us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1556730,"threads_started":1,"wal-append.queue_time_us":1356324}
I20260808 01:10:52.573557 2206 raft_consensus.cc:2454] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:52.569221 3806 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:52.580919 2201 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:10:52.580345 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:52.580552 3852 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.581539 3801 raft_consensus.cc:2781] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:10:52.582254 3801 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:52.582628 3801 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:52.576140 2172 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:52.588225 1490 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:52.588653 1568 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:52.557281 3821 raft_consensus.cc:2726] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:10:52.558756 3803 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:52.595964 1600 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:10:52.596040 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.605026 1521 raft_consensus.cc:2379] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 5: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:10:52.599393 1596 raft_consensus.cc:1226] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:52.608065 3840 raft_consensus.cc:2781] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:52.612218 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.613173 1519 raft_consensus.cc:2454] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:52.614270 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:52.615777 3873 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:10:52.616268 3873 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:52.616633 3873 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:10:52.619391 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.567618 3818 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260808 01:10:52.620080 1525 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:10:52.620270 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.620918 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:52.598542 3850 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:52.621666 3840 raft_consensus.cc:686] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:52.624125 1448 raft_consensus.cc:1261] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:10:52.602885 1489 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:52.624563 3840 consensus_queue.cc:237] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.625792 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.625957 3854 raft_consensus.cc:2781] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:52.626456 3854 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:52.626812 3854 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:52.510743 3769 consensus_queue.cc:260] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.627766 3769 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:52.629479 3873 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.630072 3765 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:52.630836 2201 raft_consensus.cc:2379] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 7: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:10:52.631348 1518 raft_consensus.cc:2379] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 4: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:10:52.625792 1408 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
W20260808 01:10:52.623029 3894 log.cc:924] Time spent T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260808 01:10:52.646821 1490 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:52.650132 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.652781 1408 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:52.659574 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.632010 3852 raft_consensus.cc:2781] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:52.663183 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:52.668226 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.670486 1444 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:52.670557 1598 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:10:52.644739 3873 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:52.672520 1408 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:52.672713 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.674512 1518 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:52.674729 1525 raft_consensus.cc:2454] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:52.681294 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.663691 3865 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.663117 3818 raft_consensus.cc:2781] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:52.684185 1408 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:52.674865 3850 raft_consensus.cc:2781] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:52.686424 1519 raft_consensus.cc:2454] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:52.576766 3706 raft_consensus.cc:348] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:52.694655 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:52.694976 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:52.695246 1444 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:52.676398 3884 raft_consensus.cc:2781] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:52.696470 1490 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:52.692166 3689 consensus_queue.cc:1035] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260808 01:10:52.690680 3748 consensus_queue.cc:1046] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260808 01:10:52.698684 3748 raft_consensus.cc:3032] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:52.699115 3748 raft_consensus.cc:729] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:52.700701 3821 raft_consensus.cc:2781] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:52.701129 3821 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:52.701551 3821 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:52.704227 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:52.704923 1600 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:52.696611 3772 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:10:52.695592 1525 raft_consensus.cc:2362] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3. Re-sending same reply.
I20260808 01:10:52.708519 2171 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:52.708606 1568 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:52.685773 3855 raft_consensus.cc:2781] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:52.684535 3852 raft_consensus.cc:686] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:52.715173 1603 raft_consensus.cc:2454] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:52.689057 3850 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:52.721254 2171 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:52.713732 1600 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:52.691682 3706 raft_consensus.cc:729] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:10:52.721101 3689 raft_consensus.cc:2726] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:10:52.744376 3706 consensus_queue.cc:260] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:52.747016 3706 ts_tablet_manager.cc:1428] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.359s user 0.006s sys 0.003s
I20260808 01:10:52.728684 3850 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:52.735702 3765 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:52.748973 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:52.725449 3884 raft_consensus.cc:686] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:52.750425 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.750818 3884 consensus_queue.cc:237] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.752125 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.716430 3865 raft_consensus.cc:2781] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:52.761509 3865 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:52.759521 1489 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:52.762001 3865 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:52.731519 3852 consensus_queue.cc:237] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:52.734297 1444 raft_consensus.cc:1226] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:52.763702 3840 raft_consensus.cc:2781] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:52.712343 3772 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:52.764506 3840 raft_consensus.cc:686] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:52.764823 3772 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:52.735116 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.766062 3689 raft_consensus.cc:2781] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:52.735023 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.765772 3840 consensus_queue.cc:237] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:10:52.766324 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 45187.4us
I20260808 01:10:52.737478 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.767674 2171 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:52.733997 3799 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.769944 3859 raft_consensus.cc:2781] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:52.770133 2171 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:52.738575 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.740583 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.741478 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.742588 3801 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.730870 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:52.753465 3748 consensus_queue.cc:260] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:52.755493 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.766697 3689 raft_consensus.cc:686] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:52.715878 3818 raft_consensus.cc:686] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:52.780218 3689 consensus_queue.cc:237] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.780359 3818 consensus_queue.cc:237] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:52.792769 3748 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:52.796231 3828 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:52.798429 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:52.799248 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.761229 3838 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.801064 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.799561 3783 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.806613 2201 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:52.773630 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:52.770730 3859 raft_consensus.cc:686] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:52.774652 1601 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:10:52.774818 1604 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:52.771574 1602 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:52.774874 3771 consensus_queue.cc:1035] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:10:52.775772 3778 raft_consensus.cc:2726] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:10:52.776672 3806 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.767680 3873 consensus_queue.cc:1046] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.010s
I20260808 01:10:52.766207 1598 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:52.778223 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:52.817248 1448 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:10:52.778326 3799 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.801834 3838 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.815707 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.820928 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.803745 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.760775 1410 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:52.729782 3855 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:52.821115 1442 raft_consensus.cc:2379] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:52.801913 1603 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:52.823159 1599 raft_consensus.cc:2454] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:52.826619 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.827153 1602 raft_consensus.cc:2454] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:52.827355 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:52.828117 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.828926 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.829540 1604 raft_consensus.cc:2454] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:52.827380 1601 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:52.842010 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.843881 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.845574 1602 raft_consensus.cc:2379] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 6: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:10:52.847065 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:52.847306 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:52.836804 3855 raft_consensus.cc:3037] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:52.816778 3803 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.860877 2201 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:10:52.875732 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.837968 1604 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:52.881161 1604 raft_consensus.cc:595] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:52.881588 1604 raft_consensus.cc:603] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:10:52.880205 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.880919 1519 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:52.881313 3769 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:52.889896 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:52.891985 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:52.892658 1601 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:52.890760 1594 raft_consensus.cc:2362] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3. Re-sending same reply.
I20260808 01:10:52.894124 1442 raft_consensus.cc:1261] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:52.860100 3854 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.908286 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260808 01:10:52.919257 3783 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.858429 3865 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.927791 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.911383 1601 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:10:52.893239 3828 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:52.929873 1489 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:52.923574 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:52.932538 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:52.928609 1519 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:52.936657 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:52.944890 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.946163 1442 raft_consensus.cc:2354] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3.
I20260808 01:10:52.946622 1596 raft_consensus.cc:2379] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 4: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:10:52.947962 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.915771 3742 consensus_queue.cc:1035] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260808 01:10:52.898444 3769 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.956849 1519 raft_consensus.cc:2396] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260808 01:10:52.963239 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:52.964233 1519 raft_consensus.cc:2454] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:52.942914 3828 raft_consensus.cc:2781] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:10:52.967610 1525 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:10:52.942687 3801 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.890102 3859 consensus_queue.cc:237] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:52.981655 1518 raft_consensus.cc:1261] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:10:52.965764 3806 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:52.997191 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:53.003865 1445 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:52.997994 3865 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:53.001775 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:10:53.002753 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 193789us
I20260808 01:10:52.980062 3884 raft_consensus.cc:2781] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:53.012591 3884 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:53.013088 3884 raft_consensus.cc:3037] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:53.013237 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:53.014117 2201 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:10:53.016486 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 255784us
I20260808 01:10:53.019225 1447 raft_consensus.cc:1261] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260808 01:10:53.019691 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 249666us
W20260808 01:10:53.023545 3900 log.cc:924] Time spent T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.131s user 0.001s sys 0.000s
I20260808 01:10:53.026667 3850 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.027153 3809 raft_consensus.cc:3032] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:10:53.027629 3809 raft_consensus.cc:729] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:53.028923 3850 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:53.028646 3809 consensus_queue.cc:260] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.030058 3809 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:52.985985 3828 raft_consensus.cc:686] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:53.033015 3828 consensus_queue.cc:237] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:53.034631 1568 leader_election.cc:398] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Vote denied by peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291) with higher term. Message: Invalid argument: T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3.
I20260808 01:10:53.034991 1568 leader_election.cc:401] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260808 01:10:53.035816 3852 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:53.042282 2201 raft_consensus.cc:2454] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:10:53.050917 3852 raft_consensus.cc:2726] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291) with higher term. Message: Invalid argument: T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3.
I20260808 01:10:53.057446 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:53.058604 1596 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:53.055087 1448 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 7.
I20260808 01:10:53.057431 1445 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:10:53.074465 1568 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:10:53.076431 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 59002.7us
I20260808 01:10:53.082259 1410 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:53.083230 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:10:53.090907 1487 leader_election.cc:334] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.661s (SENT)
I20260808 01:10:53.091457 1487 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:53.093729 3803 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:53.094166 3821 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.095083 3854 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:53.095724 3854 raft_consensus.cc:2726] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:53.097407 2171 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:53.097970 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:53.098472 1602 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:53.099715 1489 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:53.094635 3769 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:53.100132 1598 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:53.110961 1410 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:53.091709 3772 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.126920 3772 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:53.106027 1596 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:53.129374 1444 raft_consensus.cc:2379] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 6: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:53.056329 3865 consensus_queue.cc:1035] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260808 01:10:53.131016 1489 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:53.141016 1410 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:10:53.145025 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 126310us
I20260808 01:10:53.150596 1408 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:53.101514 3852 raft_consensus.cc:2781] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:53.161305 3742 consensus_queue.cc:1046] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:10:53.162971 3806 consensus_queue.cc:1046] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:10:53.148134 3838 raft_consensus.cc:2781] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:53.128638 3689 raft_consensus.cc:2741] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260808 01:10:53.168437 1410 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:53.170038 1448 raft_consensus.cc:1261] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:10:53.150159 3836 raft_consensus.cc:2781] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:53.106947 3854 raft_consensus.cc:2781] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:53.176251 1567 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:53.174837 3852 raft_consensus.cc:686] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:53.106711 3803 raft_consensus.cc:2726] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:53.175061 3772 raft_consensus.cc:2781] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:53.186214 1410 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:53.182281 3748 raft_consensus.cc:3032] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Stepping down as leader of term 4
W20260808 01:10:53.178436 3914 log.cc:924] Time spent T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
I20260808 01:10:53.191411 1408 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
W20260808 01:10:53.193517 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 96217us
I20260808 01:10:53.196231 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:53.196835 2201 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:53.190043 3748 raft_consensus.cc:729] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:53.195240 2172 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:53.182472 3854 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:53.200544 1410 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:53.180655 3836 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:53.190819 3865 consensus_queue.cc:1035] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260808 01:10:53.207293 2208 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEARNER]: Advancing to term 4
I20260808 01:10:53.184983 3821 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:53.182698 3838 raft_consensus.cc:686] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:53.207652 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:53.209803 1410 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:53.210418 2209 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:53.190239 3905 raft_consensus.cc:2741] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260808 01:10:53.190222 3799 raft_consensus.cc:2781] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:10:53.207192 3809 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:10:53.213033 2204 raft_consensus.cc:1261] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:10:53.189142 3772 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:53.193236 3803 raft_consensus.cc:2781] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 01:10:53.207623 3913 log.cc:924] Time spent T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23: Append to log took a long time: real 0.131s user 0.001s sys 0.000s
I20260808 01:10:53.202432 3854 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:53.201704 3916 raft_consensus.cc:2781] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:10:53.236002 3799 raft_consensus.cc:686] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:53.227880 3809 raft_consensus.cc:686] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:53.207768 3816 raft_consensus.cc:2741] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260808 01:10:53.211632 3836 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:53.251691 2201 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
W20260808 01:10:53.245592 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 45997.6us
I20260808 01:10:53.259595 3850 consensus_queue.cc:1035] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:10:53.260461 1567 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:53.261664 1567 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:53.261965 3783 raft_consensus.cc:3037] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:53.262590 1567 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:53.208074 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:53.263780 2205 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:53.212760 3910 raft_consensus.cc:2781] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:53.207782 3852 consensus_queue.cc:237] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.265642 3910 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:53.266098 3910 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:53.266352 3854 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.268646 3799 consensus_queue.cc:237] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.251233 3907 raft_consensus.cc:2781] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:53.270159 3907 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:53.270560 3907 raft_consensus.cc:3037] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:53.270748 2208 raft_consensus.cc:1261] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEARNER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260808 01:10:53.247664 3806 raft_consensus.cc:3032] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:53.272722 3806 raft_consensus.cc:729] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:53.273918 3861 raft_consensus.cc:2781] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:53.273700 3806 consensus_queue.cc:260] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.274459 3861 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:53.274943 3861 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:53.275240 3806 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:10:53.263139 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 50163.9us
I20260808 01:10:53.276175 3795 raft_consensus.cc:2781] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:53.276422 2171 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:53.277781 2171 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:53.278573 3873 raft_consensus.cc:2741] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260808 01:10:53.278530 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:53.279279 1598 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 4
I20260808 01:10:53.279382 3873 raft_consensus.cc:2781] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:10:53.282162 3809 consensus_queue.cc:237] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.230728 3772 raft_consensus.cc:3037] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:53.284229 3854 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:53.276592 3795 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:53.288466 3795 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:53.288920 3855 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.245041 3803 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:53.292155 3803 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:10:53.292339 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 45348.9us
I20260808 01:10:53.224486 3838 consensus_queue.cc:237] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.207350 3748 consensus_queue.cc:260] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:53.304951 1598 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:53.308911 1489 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:53.312652 1567 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:53.320767 1490 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:53.323268 2209 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
W20260808 01:10:53.326535 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 97009.4us
I20260808 01:10:53.330176 1602 raft_consensus.cc:1261] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:53.333062 1519 raft_consensus.cc:1226] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 4. Current term is 5. Ops: []
I20260808 01:10:53.336798 2206 raft_consensus.cc:1226] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 5. Ops: []
I20260808 01:10:53.307024 3916 raft_consensus.cc:686] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:53.301403 3873 raft_consensus.cc:686] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:53.341873 2208 raft_consensus.cc:1226] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:53.330389 3828 raft_consensus.cc:2781] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:53.316146 3769 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:10:53.359724 1408 consensus_peers.cc:487] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:10:53.360795 2209 raft_consensus.cc:1226] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:53.339129 3821 raft_consensus.cc:2726] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260808 01:10:53.362102 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 56338.8us
I20260808 01:10:53.362844 1567 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:53.362627 1408 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:53.363570 1567 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:53.343604 3816 raft_consensus.cc:2726] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:10:53.336916 3765 consensus_queue.cc:1046] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260808 01:10:53.390544 2209 raft_consensus.cc:1226] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:53.360656 3818 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:10:53.392386 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 60264.2us
I20260808 01:10:53.392808 1602 raft_consensus.cc:1261] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:10:53.366534 3742 raft_consensus.cc:2781] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:53.372754 3801 raft_consensus.cc:2781] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:53.386399 3828 raft_consensus.cc:686] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:53.422073 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:53.423787 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 60289us
W20260808 01:10:53.427052 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 47625.5us
I20260808 01:10:53.371366 3769 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:53.371750 3854 consensus_queue.cc:1035] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260808 01:10:53.404119 3748 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:53.405230 3742 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:53.429647 1408 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:53.434544 2209 raft_consensus.cc:2379] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 3: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:10:53.434615 2201 raft_consensus.cc:1261] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:53.446593 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:53.447252 1519 raft_consensus.cc:2362] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2. Re-sending same reply.
W20260808 01:10:53.452368 1487 leader_election.cc:334] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.991s (SENT)
I20260808 01:10:53.403098 3871 raft_consensus.cc:2781] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:10:53.423501 3801 raft_consensus.cc:686] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:53.475646 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:53.476508 2204 raft_consensus.cc:3037] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:53.476722 1518 raft_consensus.cc:1261] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:53.479878 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:53.480532 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:53.481122 1522 raft_consensus.cc:2362] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5. Re-sending same reply.
I20260808 01:10:53.478216 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:53.427989 3855 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:53.482157 1525 raft_consensus.cc:2362] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 8. Re-sending same reply.
I20260808 01:10:53.431411 3742 raft_consensus.cc:3037] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:53.402629 3818 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:53.496208 3920 log.cc:924] Time spent T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260808 01:10:53.498064 3818 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:53.500595 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:53.501096 3769 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:53.434544 3884 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:53.503471 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 67356.5us
I20260808 01:10:53.489946 3854 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:10:53.516405 3926 log.cc:924] Time spent T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20260808 01:10:53.516810 3854 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:53.431411 3852 consensus_queue.cc:1035] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260808 01:10:53.518966 3884 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:53.519033 3854 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:53.490725 3801 consensus_queue.cc:237] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.520076 1444 raft_consensus.cc:1226] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:53.454895 3871 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:53.487604 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:53.501242 1598 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:53.504190 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:53.504968 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:53.505903 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:53.524106 1522 raft_consensus.cc:2379] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 3: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:10:53.506440 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:53.524984 1518 raft_consensus.cc:2362] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2. Re-sending same reply.
I20260808 01:10:53.512189 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:53.512928 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:53.526932 2202 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:53.513854 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:53.514807 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:53.515448 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:53.516162 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:53.526232 2209 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:53.541317 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:53.518818 3795 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.429781 3915 consensus_queue.cc:1035] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260808 01:10:53.474478 3840 consensus_queue.cc:1035] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
I20260808 01:10:53.434316 3873 consensus_queue.cc:237] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.582312 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:53.596882 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:53.597010 1525 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:10:53.553014 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:53.597267 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:53.584009 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:53.551203 1489 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:53.551482 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:53.600854 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:53.604188 2209 raft_consensus.cc:2379] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:53.602023 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:53.605996 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:53.603197 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:53.583127 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:53.609419 1598 raft_consensus.cc:2454] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:53.541703 3871 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:10:53.604215 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 53103.6us
I20260808 01:10:53.611081 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
W20260808 01:10:53.612293 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 150408us
I20260808 01:10:53.550045 2204 raft_consensus.cc:2454] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:53.624408 1489 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:53.606575 2203 raft_consensus.cc:2379] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 6: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:53.632508 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:53.630474 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:53.577291 3818 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:53.635564 1598 raft_consensus.cc:1261] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260808 01:10:53.633180 1518 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:53.632524 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:53.443971 3910 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:53.643267 1408 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:53.541253 3850 consensus_queue.cc:1035] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.096s
I20260808 01:10:53.541298 3828 consensus_queue.cc:237] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.625728 3769 raft_consensus.cc:2781] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:53.592202 3836 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.634250 3933 raft_consensus.cc:2781] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:53.653934 3769 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:53.665061 3783 raft_consensus.cc:2726] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:53.665622 3931 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:10:53.667208 3840 consensus_queue.cc:1035] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:53.635751 3859 consensus_queue.cc:1046] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
W20260808 01:10:53.673197 3927 log.cc:924] Time spent T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.170s user 0.002s sys 0.000s
W20260808 01:10:53.675309 3930 log.cc:924] Time spent T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.163s user 0.000s sys 0.002s
I20260808 01:10:53.636727 3742 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.677610 3742 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:53.417390 3916 consensus_queue.cc:237] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.682742 1596 raft_consensus.cc:1226] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:53.417188 3929 raft_consensus.cc:2932] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:53.665185 3769 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:53.612069 3803 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:53.700206 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 57064.4us
W20260808 01:10:53.716485 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 69690.3us
I20260808 01:10:53.693264 3933 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:53.772591 1518 raft_consensus.cc:1226] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:53.658402 3818 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:53.780121 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:53.709753 3932 raft_consensus.cc:2781] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:53.663416 3861 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.769018 3809 raft_consensus.cc:3032] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:53.789120 1524 raft_consensus.cc:2379] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 3: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:10:53.789264 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:53.709687 3931 raft_consensus.cc:686] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
W20260808 01:10:53.782755 3907 consensus_meta.cc:287] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.512s user 0.014s sys 0.000s
I20260808 01:10:53.770536 3933 raft_consensus.cc:3037] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:53.780494 3855 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:53.810858 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:53.813702 1567 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:53.822772 1600 raft_consensus.cc:2379] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 6: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:10:53.814410 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:53.723486 3838 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20260808 01:10:53.833666 1601 raft_consensus.cc:1261] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260808 01:10:53.836437 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:53.803908 3932 raft_consensus.cc:686] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:53.840853 3932 consensus_queue.cc:237] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:53.844139 3935 log.cc:924] Time spent T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20260808 01:10:53.844373 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:53.845069 1598 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:10:53.847681 3939 log.cc:924] Time spent T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260808 01:10:53.848039 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:53.848676 1516 raft_consensus.cc:3037] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:53.852665 3910 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:53.792207 3795 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:53.831122 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:53.855190 1596 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:53.844132 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:53.856912 1594 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:53.804283 3809 raft_consensus.cc:729] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:53.782191 3772 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.865716 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:10:53.900074 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 64055.8us
W20260808 01:10:53.948516 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 94745.4us
I20260808 01:10:53.882711 1598 raft_consensus.cc:2454] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:10:53.953683 1524 raft_consensus.cc:1226] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: []
I20260808 01:10:53.847682 1534 tablet_copy_service.cc:342] P 6b09dbefd24b4d52b385c4dff5697dc5: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-d80a0f28c7c94d45b7983fc0b1386caf received from {username='slave'} at 127.0.0.1:36560
I20260808 01:10:53.956596 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:53.961648 1525 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:10:53.964967 1448 raft_consensus.cc:1226] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:53.965700 1517 raft_consensus.cc:1226] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 3. Current term is 4. Ops: []
W20260808 01:10:53.972551 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 190718us
I20260808 01:10:53.872265 3854 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:10:53.982080 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 168492us
I20260808 01:10:53.861450 1523 raft_consensus.cc:1261] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:10:53.984772 1445 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260808 01:10:53.979075 1596 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:10:53.981060 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:53.832823 3855 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:53.984652 1410 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:10:54.018414 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 44761.1us
W20260808 01:10:54.026533 2205 consensus_meta.cc:287] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.762s user 0.008s sys 0.004s
I20260808 01:10:54.027021 2205 raft_consensus.cc:2454] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:53.830092 3907 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:53.886368 3765 consensus_queue.cc:1046] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.134s
I20260808 01:10:53.981882 3836 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:54.034322 3943 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:54.032574 1408 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:10:53.984054 1534 tablet_copy_service.cc:433] P 6b09dbefd24b4d52b385c4dff5697dc5: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-d80a0f28c7c94d45b7983fc0b1386caf on tablet d80a0f28c7c94d45b7983fc0b1386caf with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:53.998184 3854 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:53.997664 3828 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:54.020944 3941 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:54.084003 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:54.085484 1442 raft_consensus.cc:1226] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:54.082683 1565 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:54.094223 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:54.028887 1567 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:10:54.103361 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 70914.7us
I20260808 01:10:54.112607 1598 raft_consensus.cc:2354] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for earlier term 5. Current term is 6.
I20260808 01:10:54.099140 1516 raft_consensus.cc:2454] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:10:54.081257 3910 raft_consensus.cc:2726] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:10:54.113896 1524 raft_consensus.cc:1226] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:54.117373 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:54.118165 1601 raft_consensus.cc:2354] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5.
I20260808 01:10:54.106463 1604 raft_consensus.cc:2454] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:54.101110 3765 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:10:54.129578 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 143608us
I20260808 01:10:54.129519 2206 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260808 01:10:54.128430 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 45869.6us
I20260808 01:10:54.133944 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:54.134706 1565 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:54.090973 3954 raft_consensus.cc:2781] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:10:54.100704 3861 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:54.091126 3803 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:54.141187 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:54.081693 3795 consensus_queue.cc:1035] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260808 01:10:54.136016 1565 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:54.136826 1601 raft_consensus.cc:1226] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:54.139047 1604 raft_consensus.cc:1226] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:54.119432 3828 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:54.644783 3821 raft_consensus.cc:3032] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:10:54.125402 3922 raft_consensus.cc:2781] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:54.244956 3950 raft_consensus.cc:2741] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20260808 01:10:54.170820 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 170893us
I20260808 01:10:54.785045 1567 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:10:54.211367 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 85661.1us
W20260808 01:10:54.787508 1487 leader_election.cc:334] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.820s (SENT)
I20260808 01:10:54.788151 1487 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:10:54.189113 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 204536us
I20260808 01:10:54.140149 3852 raft_consensus.cc:2726] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:10:54.792037 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:54.641812 1522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1830) took 1006 ms. Trace:
I20260808 01:10:54.031667 3931 consensus_queue.cc:237] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:54.269459 3971 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:10:54.137066 3765 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:54.134876 3865 raft_consensus.cc:2726] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:54.648058 1448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 613) took 1315 ms. Trace:
W20260808 01:10:54.247193 3938 log.cc:924] Time spent T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.286s user 0.001s sys 0.000s
I20260808 01:10:54.808472 1448 rpcz_store.cc:276] 0808 01:10:53.333079 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:53.333206 (+ 127us) service_pool.cc:227] Handling call
0808 01:10:53.347177 (+ 13971us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:53.965471 (+618294us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:54.647740 (+682269us) spinlock_profiling.cc:243] Waited 618 ms on lock 0x7b54001d1854. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:54.647991 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1420296576}
W20260808 01:10:54.636296 3929 consensus_meta.cc:287] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.772s user 0.007s sys 0.004s
W20260808 01:10:54.268944 3957 log.cc:924] Time spent T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.234s user 0.000s sys 0.000s
W20260808 01:10:54.454982 3956 log.cc:924] Time spent T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260808 01:10:54.104166 3873 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260808 01:10:54.835039 1521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2048) took 1502 ms. Trace:
I20260808 01:10:54.835258 3944 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:54.835525 1521 rpcz_store.cc:276] 0808 01:10:53.332420 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:53.332559 (+ 139us) service_pool.cc:227] Handling call
0808 01:10:53.350665 (+ 18106us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:10:53.350945 (+ 280us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:53.350980 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:10:53.365240 (+ 14260us) log.cc:842] Serialized 26 byte log entry
0808 01:10:53.627556 (+262316us) raft_consensus.cc:1631] Marking committed up to 1
0808 01:10:53.627589 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:53.627652 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:10:54.834820 (+1207168us) raft_consensus.cc:1671] finished
0808 01:10:54.834848 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:54.834980 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":215103,"threads_started":1,"wal-append.queue_time_us":393723}
I20260808 01:10:54.835884 3944 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:54.837509 1603 raft_consensus.cc:2932] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:54.118985 3941 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:54.268035 3960 log.cc:924] Time spent T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.184s user 0.001s sys 0.000s
I20260808 01:10:54.442978 3907 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:10:54.642035 3871 consensus_meta.cc:287] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.000s user 0.010s sys 0.002s
I20260808 01:10:54.108803 3943 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:54.685634 1594 consensus_meta.cc:287] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.819s user 0.006s sys 0.005s
I20260808 01:10:54.864835 1594 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:54.626029 3963 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:10:54.271014 3952 log.cc:924] Time spent T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.149s user 0.001s sys 0.000s
W20260808 01:10:54.453823 3936 log.cc:924] Time spent T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.242s user 0.000s sys 0.000s
I20260808 01:10:54.126302 3818 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:54.222918 3772 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:54.213846 3954 raft_consensus.cc:686] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:54.212728 1445 raft_consensus.cc:1226] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:54.059732 3809 consensus_queue.cc:260] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:54.629380 1442 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1958) took 1344 ms. Trace:
I20260808 01:10:54.876156 1442 rpcz_store.cc:276] 0808 01:10:53.285183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:53.286828 (+ 1645us) service_pool.cc:227] Handling call
0808 01:10:53.287138 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:54.109684 (+822546us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:54.629170 (+519486us) spinlock_profiling.cc:243] Waited 798 ms on lock 0x7b54002bf554. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:54.629319 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1835365120}
W20260808 01:10:54.266733 3934 log.cc:924] Time spent T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.184s user 0.000s sys 0.000s
W20260808 01:10:54.136595 3801 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.000s sys 0.011s
I20260808 01:10:54.178671 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260808 01:10:54.471438 3769 consensus_meta.cc:287] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.737s user 0.011s sys 0.000s
I20260808 01:10:54.430375 3840 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:10:54.451730 3806 consensus_meta.cc:287] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.176s user 0.006s sys 0.008s
I20260808 01:10:54.511987 3854 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:54.599202 1599 raft_consensus.cc:1226] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:54.430816 3855 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:54.501927 3838 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:54.570668 3970 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:10:54.670353 3933 consensus_meta.cc:287] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.816s user 0.009s sys 0.004s
I20260808 01:10:54.353876 1525 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 7.
W20260808 01:10:54.733925 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 597942us
I20260808 01:10:54.790923 1410 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:54.793629 1522 rpcz_store.cc:276] 0808 01:10:53.635284 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:53.635668 (+ 384us) service_pool.cc:227] Handling call
0808 01:10:53.636069 (+ 401us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:53.803117 (+167048us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:53.803144 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:53.803185 (+ 41us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:10:53.803227 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:54.641570 (+838343us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b540035cad4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:54.641609 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:54.641742 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":383655808}
I20260808 01:10:54.792060 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:54.776798 3748 raft_consensus.cc:2726] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:54.804284 3689 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:54.742559 3821 raft_consensus.cc:729] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:54.784435 3816 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:54.833382 3765 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:10:54.986646 1447 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.008s sys 0.003s
W20260808 01:10:54.987232 1447 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.550s user 0.008s sys 0.003s
I20260808 01:10:54.777200 3799 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
W20260808 01:10:54.848400 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 63447.1us
I20260808 01:10:54.989501 1567 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:54.809342 3911 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:54.808548 3922 raft_consensus.cc:686] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:54.867810 1598 raft_consensus.cc:1261] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:10:54.869131 1594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2155) took 1055 ms. Trace:
W20260808 01:10:54.855233 3742 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.828s user 0.084s sys 0.163s
I20260808 01:10:54.876812 3962 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:54.928905 3958 raft_consensus.cc:2781] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:10:54.870694 3963 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:54.939379 3941 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:54.939901 3940 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s user 0.005s sys 0.006s
I20260808 01:10:54.941983 3943 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:54.944178 1596 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:10:54.944907 3840 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:54.947508 3954 consensus_queue.cc:237] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:54.919899 3769 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:54.879556 3801 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.289s user 0.000s sys 0.013s
I20260808 01:10:54.972091 1604 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:10:54.972710 1565 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:10:54.980090 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 189282us
I20260808 01:10:55.022311 1410 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:54.982100 1525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36560 (request call id 532) took 1155 ms. Trace:
I20260808 01:10:54.919626 3818 raft_consensus.cc:2726] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:10:54.867409 3871 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:54.985739 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:10:55.010730 3940 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.987s user 0.005s sys 0.006s
I20260808 01:10:55.037096 1567 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:54.833693 1519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1829) took 1205 ms. Trace:
I20260808 01:10:54.994413 1594 rpcz_store.cc:276] 0808 01:10:53.809684 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:53.809830 (+ 146us) service_pool.cc:227] Handling call
0808 01:10:54.865443 (+1055613us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:10:55.042202 1488 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.979s user 0.009s sys 0.010s
W20260808 01:10:55.042637 1488 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.980s user 0.009s sys 0.010s
I20260808 01:10:55.042800 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:10:54.987905 1600 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.002s sys 0.009s
I20260808 01:10:55.046892 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:54.982750 3838 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:55.013015 3828 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:54.996407 1408 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.773s user 0.004s sys 0.042s
I20260808 01:10:54.985028 3855 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:54.985205 2203 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 685) took 1215 ms. Trace:
I20260808 01:10:55.081993 1597 raft_consensus.cc:1226] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 5. Current term is 6. Ops: []
W20260808 01:10:54.996620 3742 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.977s user 0.084s sys 0.163s
I20260808 01:10:55.010032 2201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 686) took 1231 ms. Trace:
I20260808 01:10:55.048929 3944 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:55.084247 2201 rpcz_store.cc:276] 0808 01:10:53.778334 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:53.778616 (+ 282us) service_pool.cc:227] Handling call
0808 01:10:53.790283 (+ 11667us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:10:53.792971 (+ 2688us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:53.792999 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:10:53.793447 (+ 448us) log.cc:842] Serialized 26 byte log entry
0808 01:10:53.864261 (+ 70814us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:10:53.864300 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:53.864409 (+ 109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:10:54.270153 (+405744us) spinlock_profiling.cc:243] Waited 81.1 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:10:54.996575 (+726422us) raft_consensus.cc:1671] finished
0808 01:10:54.996605 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:55.009925 (+ 13320us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2178,"raft.queue_time_us":3389,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"spinlock_wait_cycles":186449792,"thread_start_us":70381,"threads_started":1,"wal-append.queue_time_us":229678}
W20260808 01:10:54.993913 1438 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.006s sys 0.006s
W20260808 01:10:55.086658 1438 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.628s user 0.007s sys 0.006s
I20260808 01:10:55.027051 3965 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:10:55.018330 3916 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.759s user 0.006s sys 0.004s
I20260808 01:10:55.088246 3769 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:55.088616 3916 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.835s user 0.006s sys 0.004s
I20260808 01:10:55.047582 3861 raft_consensus.cc:2726] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:10:55.028084 3976 log.cc:924] Time spent T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.119s user 0.001s sys 0.000s
I20260808 01:10:55.090044 1594 raft_consensus.cc:1261] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:10:55.092367 3838 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:55.080365 3821 consensus_queue.cc:260] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:55.094925 3821 raft_consensus.cc:3037] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:10:55.023702 1525 rpcz_store.cc:276] 0808 01:10:53.815024 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:53.859271 (+ 44247us) service_pool.cc:227] Handling call
0808 01:10:54.970818 (+1111547us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:55.088315 3965 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:55.101162 3965 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:55.105831 3818 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:55.106406 3818 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:55.108664 3818 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:55.109809 3838 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:55.110284 3838 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:55.110730 3769 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:55.111274 3769 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:55.006634 3799 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:55.030720 1489 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:55.051648 3963 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:10:54.987574 1411 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.000s sys 0.011s
W20260808 01:10:55.117183 1411 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.739s user 0.000s sys 0.012s
I20260808 01:10:55.028645 3967 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:55.040040 1519 rpcz_store.cc:276] 0808 01:10:53.628542 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:53.628681 (+ 139us) service_pool.cc:227] Handling call
0808 01:10:53.629011 (+ 330us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:10:53.629313 (+ 302us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:53.629351 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:10:53.649912 (+ 20561us) log.cc:842] Serialized 26 byte log entry
0808 01:10:53.982216 (+332304us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:10:54.201763 (+219547us) log.cc:842] Serialized 14 byte log entry
0808 01:10:54.201982 (+ 219us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:54.202076 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:10:54.833407 (+631331us) raft_consensus.cc:1671] finished
0808 01:10:54.833437 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:54.833616 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54169,"prepare.queue_time_us":453094,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":56,"raft.queue_time_us":46212,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":13344,"thread_start_us":449007,"threads_started":2,"wal-append.queue_time_us":355450}
I20260808 01:10:55.049906 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:55.037972 3962 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:55.038195 3795 raft_consensus.cc:2726] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:10:55.035310 3958 raft_consensus.cc:686] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
W20260808 01:10:54.986421 3803 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.706s user 0.008s sys 0.004s
I20260808 01:10:55.130138 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:55.126551 1599 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
W20260808 01:10:55.073328 1518 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.050s user 0.021s sys 0.035s
W20260808 01:10:55.132400 1518 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.109s user 0.021s sys 0.035s
W20260808 01:10:55.080933 1408 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.857s user 0.005s sys 0.050s
W20260808 01:10:55.133076 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 912065us
W20260808 01:10:55.081223 1600 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s user 0.004s sys 0.014s
I20260808 01:10:54.981887 3933 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:55.072178 1567 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:10:55.030297 1410 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:55.044631 3748 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:10:54.996129 3689 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:54.992553 3816 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:10:55.116173 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 85545.5us
W20260808 01:10:55.033898 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 61305.1us
I20260808 01:10:55.149842 1565 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:54.982944 3970 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:55.007799 3911 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:55.135252 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 105050us
I20260808 01:10:55.136466 1442 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:10:55.116775 3941 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:10:55.138959 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 351469us
I20260808 01:10:55.112303 3838 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:55.130826 1439 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:10:55.123103 3916 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:55.036042 3943 consensus_queue.cc:1035] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
I20260808 01:10:55.179782 3873 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:55.179899 3806 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260808 01:10:55.193117 3873 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:55.194157 3916 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:55.195377 3873 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:55.197124 3840 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:55.157811 1489 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:10:55.159711 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 114313us
I20260808 01:10:55.161835 1440 raft_consensus.cc:1226] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:55.199138 1567 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:10:55.136977 3803 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.856s user 0.008s sys 0.004s
I20260808 01:10:55.162384 1410 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:10:55.208878 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 51172.3us
I20260808 01:10:55.164170 2205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 691) took 1043 ms. Trace:
I20260808 01:10:55.131721 3828 raft_consensus.cc:2781] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:10:55.169796 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:55.127487 3940 consensus_queue.cc:1046] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260808 01:10:55.160317 3769 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:55.144743 3801 raft_consensus.cc:2781] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 01:10:55.168289 2171 connection.cc:232] RPC call timeout handler was delayed by -1.1085161750088446s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22260000000000002s for a response
I20260808 01:10:55.197202 3911 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:55.151404 1565 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:55.183416 3838 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:55.156163 3944 raft_consensus.cc:2741] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260808 01:10:55.160456 3861 raft_consensus.cc:2726] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:55.220746 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
W20260808 01:10:55.222167 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 59910.4us
I20260808 01:10:55.169242 3748 raft_consensus.cc:686] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 8 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:55.228889 1448 raft_consensus.cc:1226] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:55.082240 2203 rpcz_store.cc:276] 0808 01:10:53.768822 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:53.773225 (+ 4403us) service_pool.cc:227] Handling call
0808 01:10:53.851888 (+ 78663us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:10:53.852296 (+ 408us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:53.852327 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:10:53.852894 (+ 567us) log.cc:842] Serialized 26 byte log entry
0808 01:10:54.028041 (+175147us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:10:54.031105 (+ 3064us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:54.060010 (+ 28905us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:10:54.954574 (+894564us) raft_consensus.cc:1671] finished
0808 01:10:54.954594 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:54.983847 (+ 29253us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2116,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":12440,"spinlock_wait_cycles":1937152,"thread_start_us":145897,"threads_started":1,"wal-append.queue_time_us":164066}
I20260808 01:10:55.217990 3840 raft_consensus.cc:2781] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:55.222609 3828 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:55.242379 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:55.248304 1599 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:10:55.249605 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 98302.6us
I20260808 01:10:55.250651 1565 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:55.242477 1567 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:55.228017 3772 raft_consensus.cc:2781] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:55.245015 1410 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:55.222784 3795 raft_consensus.cc:2726] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 01:10:55.206300 1487 leader_election.cc:334] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.615s (SENT)
I20260808 01:10:55.272867 1487 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:55.269436 1565 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:55.258276 3828 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:55.223065 3801 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:55.271066 3916 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:55.267225 3742 raft_consensus.cc:2726] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:55.214097 2205 rpcz_store.cc:276] 0808 01:10:54.121126 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:54.125961 (+ 4835us) service_pool.cc:227] Handling call
0808 01:10:54.187433 (+ 61472us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:55.139381 (+951948us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:55.144994 (+ 5613us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:55.163691 (+ 18697us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:10:55.163773 (+ 82us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:55.163884 (+ 111us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:55.164083 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":503600,"raft.queue_time_us":571072,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":32216,"thread_start_us":323622,"threads_started":1}
I20260808 01:10:55.286207 3916 raft_consensus.cc:2781] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:10:55.286760 3916 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:55.287207 3916 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:55.289316 3855 raft_consensus.cc:2726] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:10:55.118897 3922 consensus_queue.cc:237] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:55.272113 3772 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:55.218091 3873 consensus_queue.cc:1046] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.031s
I20260808 01:10:55.275800 1410 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:55.242156 3932 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:55.247520 3840 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:55.266047 3748 consensus_queue.cc:237] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:55.266458 3861 raft_consensus.cc:2726] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:55.180778 3941 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:55.238874 3838 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:55.308008 3929 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:55.310588 3865 raft_consensus.cc:2726] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260808 01:10:55.313395 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 107135us
W20260808 01:10:55.296168 3968 log.cc:924] Time spent T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20260808 01:10:55.354260 1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 505) took 1221 ms. Trace:
I20260808 01:10:55.360116 1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 499) took 1276 ms. Trace:
I20260808 01:10:55.285003 3801 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:55.338708 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:55.342167 3840 raft_consensus.cc:3037] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:55.311668 3772 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:55.239264 3990 consensus_queue.cc:1046] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.079s
W20260808 01:10:55.345073 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 69371.5us
I20260808 01:10:55.297607 3742 raft_consensus.cc:2726] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:10:55.350304 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 113958us
I20260808 01:10:55.393656 1567 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:55.314018 3769 raft_consensus.cc:2726] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:55.361074 1596 rpcz_store.cc:276] 0808 01:10:54.083498 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:54.083684 (+ 186us) service_pool.cc:227] Handling call
0808 01:10:55.359907 (+1276223us) spinlock_profiling.cc:243] Waited 835 ms on lock 0x7b5400120f54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:55.360047 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1919494912}
W20260808 01:10:55.364606 1487 leader_election.cc:334] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.878s (SENT)
I20260808 01:10:55.395712 1487 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:55.280637 3871 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:55.359953 1600 rpcz_store.cc:276] 0808 01:10:54.132704 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:54.133087 (+ 383us) service_pool.cc:227] Handling call
0808 01:10:54.133476 (+ 389us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:10:54.172659 (+ 39183us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:54.172708 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:10:54.210654 (+ 37946us) log.cc:842] Serialized 26 byte log entry
0808 01:10:55.134416 (+923762us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:10:55.324455 (+190039us) log.cc:842] Serialized 14 byte log entry
0808 01:10:55.353584 (+ 29129us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:55.353728 (+ 144us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:10:55.353803 (+ 75us) raft_consensus.cc:1671] finished
0808 01:10:55.353845 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:55.354134 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28313,"raft.queue_time_us":31728,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":32056,"thread_start_us":950087,"threads_started":2,"wal-append.queue_time_us":860865}
W20260808 01:10:55.372407 3975 log.cc:924] Time spent T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20260808 01:10:55.364228 1444 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:10:55.377928 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:55.380869 3996 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:55.381701 3962 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:55.384142 3992 consensus_queue.cc:1046] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:10:55.385201 3689 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:55.385686 3816 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:55.358117 3799 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:55.349318 3979 raft_consensus.cc:2781] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:55.357950 3929 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:55.442276 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 48669.7us
I20260808 01:10:55.445009 1567 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:55.390609 1410 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:55.358678 3933 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:55.405340 1568 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:10:55.406723 1487 leader_election.cc:334] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.117s (SENT)
W20260808 01:10:55.453735 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 47058.8us
I20260808 01:10:55.276113 3958 consensus_queue.cc:237] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:55.344458 3932 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:55.420495 1448 raft_consensus.cc:1226] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 01:10:55.422281 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:55.431753 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:55.418027 3940 raft_consensus.cc:2726] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:10:55.476095 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 85620.7us
I20260808 01:10:55.451017 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:55.428845 3769 raft_consensus.cc:2726] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:10:55.429994 3962 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:55.472734 1601 raft_consensus.cc:3037] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:55.472452 1596 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:55.457468 1447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1963) took 1098 ms. Trace:
I20260808 01:10:55.473234 1518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1832) took 1661 ms. Trace:
I20260808 01:10:55.432447 3979 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:55.476920 1410 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:55.472412 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:10:55.478287 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 73085.5us
I20260808 01:10:55.481235 1518 rpcz_store.cc:276] 0808 01:10:53.812327 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:53.812442 (+ 115us) service_pool.cc:227] Handling call
0808 01:10:53.836622 (+ 24180us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:10:53.886252 (+ 49630us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:53.946703 (+ 60451us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:10:53.960259 (+ 13556us) log.cc:842] Serialized 26 byte log entry
0808 01:10:55.157107 (+1196848us) raft_consensus.cc:1631] Marking committed up to 3
0808 01:10:55.157154 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:55.157240 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:10:55.473015 (+315775us) raft_consensus.cc:1671] finished
0808 01:10:55.473036 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:55.473171 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48275,"raft.queue_time_us":27374,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":81529,"thread_start_us":1109151,"threads_started":1,"wal-append.queue_time_us":1223190}
I20260808 01:10:55.480397 1447 rpcz_store.cc:276] 0808 01:10:54.359181 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:54.359356 (+ 175us) service_pool.cc:227] Handling call
0808 01:10:54.359730 (+ 374us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:54.360001 (+ 271us) raft_consensus.cc:1520] Early marking committed up to index 2
0808 01:10:54.990816 (+630815us) log.cc:842] Serialized 14 byte log entry
0808 01:10:55.456995 (+466179us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:55.457066 (+ 71us) raft_consensus.cc:1631] Marking committed up to 2
0808 01:10:55.457136 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:55.457223 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:55.457376 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1581,"prepare.queue_time_us":736085,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":44962,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"thread_start_us":1015604,"threads_started":3,"wal-append.queue_time_us":440051}
W20260808 01:10:55.486068 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 194743us
I20260808 01:10:55.399802 3999 consensus_queue.cc:1035] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20260808 01:10:55.488739 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:55.466360 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:55.437877 3987 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:55.491241 2171 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:55.445957 3970 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:55.504877 1410 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:55.504876 2171 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:55.506722 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:55.486696 3979 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:55.423141 3996 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:55.482591 3991 consensus_queue.cc:1046] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.146s
I20260808 01:10:55.624696 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:55.790428 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 285047us
I20260808 01:10:55.485823 3962 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:55.494736 3838 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:55.505352 3954 raft_consensus.cc:2781] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 01:10:55.703637 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 201647us
I20260808 01:10:55.708108 1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 508) took 1483 ms. Trace:
I20260808 01:10:55.485165 3783 raft_consensus.cc:2781] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:10:55.709551 3915 raft_consensus.cc:2726] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260808 01:10:55.709604 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 204792us
I20260808 01:10:55.710216 3931 raft_consensus.cc:2726] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:10:55.827679 1410 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:55.829021 1410 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:10:55.713649 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 208842us
I20260808 01:10:55.724848 1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2421) took 1017 ms. Trace:
I20260808 01:10:55.727031 3802 raft_consensus.cc:2726] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:55.729017 3778 raft_consensus.cc:2726] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:55.730542 3803 raft_consensus.cc:1050] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:55.829830 1603 rpcz_store.cc:276] 0808 01:10:54.707234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:54.707377 (+ 143us) service_pool.cc:227] Handling call
0808 01:10:54.777053 (+ 69676us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:54.837372 (+ 60319us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 01:10:54.880671 (+ 43299us) log.cc:842] Serialized 14 byte log entry
0808 01:10:55.198905 (+318234us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:55.198953 (+ 48us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:10:55.199000 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:55.724522 (+525522us) spinlock_profiling.cc:243] Waited 60.1 ms on lock 0x7b540018d8d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:55.724593 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:55.724764 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6884,"raft.queue_time_us":6721,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":14208,"spinlock_wait_cycles":138188672,"thread_start_us":300873,"threads_started":1,"wal-append.queue_time_us":410852}
I20260808 01:10:55.732977 3806 raft_consensus.cc:3032] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:55.736011 2201 raft_consensus.cc:1261] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:10:55.739537 2209 raft_consensus.cc:1261] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:10:55.741901 1446 raft_consensus.cc:1226] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:55.743757 1342 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55756 (request call id 336) took 3381 ms. Trace:
I20260808 01:10:55.755383 3932 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:10:55.756631 3997 log.cc:924] Time spent T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.413s user 0.002s sys 0.005s
I20260808 01:10:55.761128 3855 raft_consensus.cc:1050] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:10:55.715701 1601 raft_consensus.cc:2454] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:55.824348 3954 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:55.497913 3854 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:10:55.771080 4000 log.cc:924] Time spent T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.280s user 0.000s sys 0.006s
W20260808 01:10:55.777096 4001 log.cc:924] Time spent T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.284s user 0.004s sys 0.002s
I20260808 01:10:55.478664 3933 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:10:55.778537 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 287273us
I20260808 01:10:55.923190 1567 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:10:55.923759 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 94814us
I20260808 01:10:55.533185 3941 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:55.814685 1600 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:10:55.813685 1597 rpcz_store.cc:276] 0808 01:10:54.224458 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:54.224608 (+ 150us) service_pool.cc:227] Handling call
0808 01:10:54.224923 (+ 315us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:55.082448 (+857525us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:55.616225 (+533777us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b54002dd2d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:55.708032 (+ 91807us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1969599744}
I20260808 01:10:55.825078 1489 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:55.703462 3689 raft_consensus.cc:3032] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:55.813107 3801 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:55.550364 1438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1964) took 1189 ms. Trace:
I20260808 01:10:55.768863 3929 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:55.911135 3954 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:10:55.862489 3803 raft_consensus.cc:1054] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d: Unable to promote non-voter 95b661b03f1d460b9657e61d4b0247cd: Illegal state: Leader has not yet committed an operation in its own term
I20260808 01:10:55.806399 3816 consensus_queue.cc:1046] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.087s
I20260808 01:10:55.862129 3838 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:55.851486 3806 raft_consensus.cc:729] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:55.854764 3795 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:55.925166 1410 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:10:55.980058 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 54984.3us
I20260808 01:10:55.981074 1410 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:10:55.936913 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 111978us
I20260808 01:10:55.504492 3987 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:10:55.909612 3855 raft_consensus.cc:1054] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d: Unable to promote non-voter 95b661b03f1d460b9657e61d4b0247cd: Illegal state: Leader has not yet committed an operation in its own term
I20260808 01:10:55.910308 3915 raft_consensus.cc:2781] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:55.827412 3742 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:55.914083 3970 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:55.913884 3963 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:55.926877 3931 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:55.931526 3783 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:10:55.999861 1599 consensus_meta.cc:287] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.733s user 0.010s sys 0.003s
I20260808 01:10:55.875671 1342 rpcz_store.cc:276] 0808 01:10:52.363361 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:52.366740 (+ 3379us) service_pool.cc:227] Handling call
0808 01:10:55.723803 (+3357063us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 01:10:52.716608 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:10:52.717281 (+ 673us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:10:52.717339 (+ 58us) write_op.cc:435] Acquired schema lock
0808 01:10:52.717373 (+ 34us) tablet.cc:660] Decoding operations
0808 01:10:52.717912 (+ 539us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:10:52.718011 (+ 99us) write_op.cc:641] Partition lock acquired for write op
0808 01:10:52.718039 (+ 28us) tablet.cc:683] Acquiring locks for 2 operations
0808 01:10:52.718554 (+ 515us) tablet.cc:699] Row locks acquired
0808 01:10:52.718585 (+ 31us) write_op.cc:260] PREPARE: finished
0808 01:10:52.718785 (+ 200us) write_op.cc:270] Start()
0808 01:10:52.718962 (+ 177us) write_op.cc:276] Timestamp: P: 1786151452718699 usec, L: 0
0808 01:10:52.719010 (+ 48us) op_driver.cc:331] REPLICATION: starting
0808 01:10:52.726151 (+ 7141us) log.cc:842] Serialized 924 byte log entry
0808 01:10:53.023061 (+296910us) spinlock_profiling.cc:243] Waited 33.5 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe349af95fe 00007fe352930a39 00007fe349b168bb 00007fe349b14cbf 00007fe349b16e22 00007fe34c6c05bb
0808 01:10:53.503135 (+480074us) op_driver.cc:444] REPLICATION: finished
0808 01:10:53.865368 (+362233us) write_op.cc:301] APPLY: starting
0808 01:10:53.948388 (+ 83020us) tablet.cc:1365] starting BulkCheckPresence
0808 01:10:53.985673 (+ 37285us) tablet.cc:1368] finished BulkCheckPresence
0808 01:10:54.020849 (+ 35176us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 01:10:54.169907 (+149058us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 01:10:54.357753 (+187846us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0808 01:10:54.435555 (+ 77802us) write_op.cc:312] APPLY: finished
0808 01:10:54.458475 (+ 22920us) log.cc:842] Serialized 29 byte log entry
0808 01:10:55.009368 (+550893us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:10:55.013433 (+ 4065us) write_op.cc:454] Released schema lock
0808 01:10:55.014962 (+ 1529us) write_op.cc:341] FINISH: Updating metrics
0808 01:10:55.082659 (+ 67697us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:10:55.083712 (+ 1053us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":245280,"apply.run_cpu_time_us":17777,"apply.run_wall_time_us":1254850,"num_ops":2,"prepare.queue_time_us":135799,"prepare.run_cpu_time_us":30309,"prepare.run_wall_time_us":359141,"raft.queue_time_us":136280,"raft.run_cpu_time_us":9648,"raft.run_wall_time_us":282073,"replication_time_us":764611,"spinlock_wait_cycles":76958464,"thread_start_us":1025432,"threads_started":5,"wal-append.queue_time_us":1023516,"wal-append.run_cpu_time_us":26338,"wal-append.run_wall_time_us":1281433}]]}
I20260808 01:10:55.955397 3818 raft_consensus.cc:2781] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:56.034592 3818 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:56.035476 3818 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:56.019789 3970 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:56.038481 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:56.024178 3931 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:55.937763 3689 raft_consensus.cc:729] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:56.041158 3996 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:55.973002 3748 raft_consensus.cc:2781] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:56.041785 3931 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:56.042315 3748 raft_consensus.cc:686] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:56.041428 3689 consensus_queue.cc:260] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:56.024305 3915 raft_consensus.cc:686] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:56.044919 1338 catalog_manager.cc:5495] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 2 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:56.012490 1489 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:56.024639 3783 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:56.046778 1338 catalog_manager.cc:5495] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 2 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:56.005291 3795 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:56.008641 3929 consensus_queue.cc:1035] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260808 01:10:56.018745 3742 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:55.915302 3932 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
I20260808 01:10:56.045312 3915 consensus_queue.cc:237] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:55.927964 3922 consensus_queue.cc:1035] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.088s
I20260808 01:10:56.058944 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:56.060876 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:55.914302 3854 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:56.065898 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:56.066548 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:56.085227 1446 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:56.031785 1599 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:56.073328 1338 catalog_manager.cc:5495] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 2 to 3, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 3 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:10:56.051491 3929 raft_consensus.cc:2781] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:10:55.972611 3907 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:56.078974 1594 raft_consensus.cc:2379] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 4: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:10:56.061762 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:56.004925 3803 raft_consensus.cc:2726] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:10:56.025709 3855 raft_consensus.cc:2741] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260808 01:10:55.942869 3933 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:56.095531 1338 catalog_manager.cc:5495] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 1 to 2, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 2 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:56.055425 3689 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:56.078081 1601 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:10:56.008241 3865 consensus_queue.cc:1035] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.110s
I20260808 01:10:56.032905 3991 consensus_queue.cc:1046] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.088s
I20260808 01:10:56.009753 3838 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:56.052909 3963 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:56.246248 1442 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 620) took 1068 ms. Trace:
W20260808 01:10:56.326053 2170 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.010s sys 0.023s
W20260808 01:10:56.552312 2170 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.807s user 0.019s sys 0.027s
W20260808 01:10:56.268049 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 241837us
I20260808 01:10:56.255752 4015 consensus_queue.cc:1046] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.581s
I20260808 01:10:56.554983 4015 consensus_queue.cc:1225] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer 6b09dbefd24b4d52b385c4dff5697dc5 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260808 01:10:56.555671 4014 raft_consensus.cc:3032] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:56.555734 2204 raft_consensus.cc:1261] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:10:56.556255 4014 raft_consensus.cc:729] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:56.183553 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:10:56.533790 2208 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.005s sys 0.012s
W20260808 01:10:56.562458 2208 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.670s user 0.005s sys 0.013s
W20260808 01:10:56.295311 1487 leader_election.cc:334] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.507s (SENT)
W20260808 01:10:56.143378 1490 leader_election.cc:334] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.631s (SENT)
W20260808 01:10:56.563776 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 420431us
W20260808 01:10:56.563997 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 268736us
I20260808 01:10:56.120324 1489 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:10:56.459138 2171 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.000s sys 0.010s
W20260808 01:10:56.568449 2171 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.705s user 0.000s sys 0.013s
I20260808 01:10:56.076561 3795 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:56.490765 2200 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.685s user 0.017s sys 0.037s
W20260808 01:10:56.578967 2200 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.774s user 0.018s sys 0.038s
W20260808 01:10:56.371428 2206 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.005s sys 0.026s
W20260808 01:10:56.579577 2206 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.756s user 0.016s sys 0.026s
I20260808 01:10:56.087036 3996 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:10:56.389699 3859 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.012s sys 0.017s
I20260808 01:10:56.591554 1440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 617) took 1457 ms. Trace:
I20260808 01:10:56.591740 1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2434) took 1438 ms. Trace:
I20260808 01:10:56.592115 1600 rpcz_store.cc:276] 0808 01:10:55.153502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:55.424857 (+271355us) service_pool.cc:227] Handling call
0808 01:10:56.591584 (+1166727us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b5400121954. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:56.591685 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":745256832}
I20260808 01:10:56.558104 1338 catalog_manager.cc:5495] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 2 to 3, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 3 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:56.575320 1446 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:56.582389 3987 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:10:56.543723 4017 log.cc:924] Time spent T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.436s user 0.001s sys 0.001s
W20260808 01:10:56.193907 3772 consensus_meta.cc:287] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.797s user 0.007s sys 0.007s
W20260808 01:10:56.585421 3840 consensus_meta.cc:287] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 1.192s user 0.006s sys 0.006s
W20260808 01:10:56.585435 3783 consensus_meta.cc:287] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.539s user 0.011s sys 0.000s
I20260808 01:10:56.586170 2205 raft_consensus.cc:1261] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:10:56.147593 3748 consensus_queue.cc:237] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:56.580397 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 697) took 1346 ms. Trace:
I20260808 01:10:56.677006 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:10:56.234567 3967 consensus_meta.cc:287] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 1.097s user 0.012s sys 0.007s
W20260808 01:10:56.194712 4016 log.cc:924] Time spent T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
W20260808 01:10:56.365157 3916 consensus_meta.cc:287] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 1.077s user 0.017s sys 0.000s
W20260808 01:10:56.172737 4002 log.cc:924] Time spent T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.331s user 0.000s sys 0.007s
I20260808 01:10:56.676934 2206 rpcz_store.cc:276] 0808 01:10:55.234004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:55.234248 (+ 244us) service_pool.cc:227] Handling call
0808 01:10:55.247254 (+ 13006us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:55.247577 (+ 323us) raft_consensus.cc:1520] Early marking committed up to index 7
0808 01:10:55.478648 (+231071us) log.cc:842] Serialized 14 byte log entry
0808 01:10:56.580063 (+1101415us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:56.580123 (+ 60us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:10:56.580167 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:56.580228 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:56.580345 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15000,"prepare.queue_time_us":553043,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":91049,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":14843,"thread_start_us":1314893,"threads_started":3,"wal-append.queue_time_us":690563}
I20260808 01:10:56.551709 1442 rpcz_store.cc:276] 0808 01:10:55.177730 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:55.207511 (+ 29781us) service_pool.cc:227] Handling call
0808 01:10:55.234813 (+ 27302us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:10:55.433282 (+198469us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:55.433311 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:10:55.441637 (+ 8326us) log.cc:842] Serialized 26 byte log entry
0808 01:10:55.867145 (+425508us) raft_consensus.cc:1631] Marking committed up to 8
0808 01:10:55.867196 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:55.867291 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:10:56.245907 (+378616us) raft_consensus.cc:1671] finished
0808 01:10:56.245930 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:56.246155 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5044,"raft.queue_time_us":34844,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":4247,"thread_start_us":567106,"threads_started":2,"wal-append.queue_time_us":548043}
W20260808 01:10:56.491021 3821 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.735s user 0.024s sys 0.034s
I20260808 01:10:56.544121 3990 raft_consensus.cc:3032] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:56.591981 1440 rpcz_store.cc:276] 0808 01:10:55.133848 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:55.133990 (+ 142us) service_pool.cc:227] Handling call
0808 01:10:55.161676 (+ 27686us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:55.198964 (+ 37288us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:56.591351 (+1392387us) spinlock_profiling.cc:243] Waited 6.44 ms on lock 0x7b54001be4d0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:56.591488 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14814464}
I20260808 01:10:56.473640 3809 raft_consensus.cc:3037] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:56.687830 3967 raft_consensus.cc:2726] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:10:56.719146 1445 raft_consensus.cc:1226] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:56.720146 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:56.137897 4009 raft_consensus.cc:2726] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:10:56.434792 3742 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:56.465205 3970 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:56.172540 3941 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:56.721015 1440 raft_consensus.cc:2454] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:56.565006 3801 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:56.663380 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:56.721344 3970 raft_consensus.cc:2781] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:56.721776 1439 raft_consensus.cc:2362] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3. Re-sending same reply.
I20260808 01:10:56.721904 3970 raft_consensus.cc:686] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:56.578907 3922 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:56.722680 3922 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:56.721354 3941 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:56.311579 3940 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:56.204061 4010 raft_consensus.cc:2726] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:10:56.587291 3861 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:10:56.590333 3859 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.777s user 0.020s sys 0.020s
I20260808 01:10:56.678002 1446 raft_consensus.cc:2379] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 3: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
W20260808 01:10:56.664098 4018 log.cc:924] Time spent T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.469s user 0.002s sys 0.000s
I20260808 01:10:56.594292 1338 catalog_manager.cc:5495] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d reported cstate change: term changed from 3 to 4, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 4 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:10:56.734366 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:56.619037 3772 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:56.252028 3943 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.193s sys 0.215s
I20260808 01:10:56.687315 3795 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:56.682708 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:56.720453 4036 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:56.593300 3974 consensus_queue.cc:1035] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.118s
I20260808 01:10:56.194150 3962 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:10:56.713759 3821 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.955s user 0.026s sys 0.035s
W20260808 01:10:56.746809 3943 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.032s user 0.255s sys 0.301s
W20260808 01:10:56.568696 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 705539us
I20260808 01:10:56.639210 3840 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:56.710474 2205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 700) took 1230 ms. Trace:
I20260808 01:10:56.119939 3907 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:56.734230 3940 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:56.756505 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:56.738396 3861 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:56.759500 3838 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:56.713666 3990 raft_consensus.cc:729] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:56.800370 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:56.760198 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 52589.7us
I20260808 01:10:56.728245 3854 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:56.676234 3783 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:56.769217 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:56.731067 3941 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:56.818274 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:56.602130 3996 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:56.780436 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:56.829219 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:56.832636 1446 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:10:56.832623 1597 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:56.836975 1597 raft_consensus.cc:595] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:56.837455 1597 raft_consensus.cc:603] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:10:56.838685 3855 raft_consensus.cc:1050] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
W20260808 01:10:56.839236 3855 raft_consensus.cc:1054] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d: Unable to promote non-voter 95b661b03f1d460b9657e61d4b0247cd: Illegal state: Leader has not yet committed an operation in its own term
I20260808 01:10:56.840236 3963 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:56.768203 3799 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.884s user 0.011s sys 0.044s
W20260808 01:10:56.841174 3799 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.979s user 0.012s sys 0.044s
I20260808 01:10:56.808527 3859 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:56.761552 3987 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:10:56.782680 1566 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.861s user 0.007s sys 0.016s
W20260808 01:10:56.776077 1567 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.839s user 0.012s sys 0.010s
W20260808 01:10:56.803434 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 683785us
I20260808 01:10:56.707551 3916 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:56.795657 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:56.663455 4014 consensus_queue.cc:260] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:56.800508 3828 consensus_meta.cc:287] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 1.504s user 0.015s sys 0.001s
W20260808 01:10:56.792098 3929 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.001s sys 0.017s
I20260808 01:10:56.823920 1440 raft_consensus.cc:2379] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 4: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:56.738646 3992 raft_consensus.cc:3032] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:56.764741 3795 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:56.836280 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:56.788302 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:56.739732 3922 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:56.777024 4036 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:56.853096 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:56.854003 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:56.854004 1489 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:56.856005 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:56.777580 3970 consensus_queue.cc:237] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:56.851460 1566 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.930s user 0.008s sys 0.016s
I20260808 01:10:56.876420 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
W20260808 01:10:56.851758 1567 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.915s user 0.013s sys 0.011s
I20260808 01:10:56.911638 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:10:57.063019 1603 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.945s user 0.051s sys 0.041s
W20260808 01:10:57.242875 1603 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.125s user 0.051s sys 0.051s
I20260808 01:10:57.238926 3742 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:56.788128 2205 rpcz_store.cc:276] 0808 01:10:55.474910 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:55.480223 (+ 5313us) service_pool.cc:227] Handling call
0808 01:10:56.585681 (+1105458us) spinlock_profiling.cc:243] Waited 5.55 ms on lock 0x7b5800005198. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adfc6c 00007fe349adfbd6 000000000034aea9 00007fe35190ba30 00007fe3519d17b4 00007fe3519d1821 00007fe3519952b0 00007fe3519807f9 00007fe3519802a2 00007fe34b066684
0808 01:10:56.585818 (+ 137us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:56.677607 (+ 91789us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:56.705545 (+ 27938us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12772096}
W20260808 01:10:57.241464 2208 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.003s sys 0.024s
W20260808 01:10:57.315811 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 461846us
I20260808 01:10:57.317194 1489 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:56.894235 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
W20260808 01:10:56.896642 1565 leader_election.cc:334] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.600s (SENT)
W20260808 01:10:57.320448 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 423837us
W20260808 01:10:57.315529 2208 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.752s user 0.003s sys 0.024s
W20260808 01:10:57.236465 4037 log.cc:924] Time spent T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.444s user 0.001s sys 0.000s
I20260808 01:10:57.239210 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260808 01:10:56.864040 1487 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.682s (SENT) [suppressed 48 similar messages]
W20260808 01:10:57.328238 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 464217us
I20260808 01:10:57.328439 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:56.900720 1448 raft_consensus.cc:1226] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: []
I20260808 01:10:56.902143 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:10:57.237886 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 1.31468e+06us
W20260808 01:10:57.245851 1490 leader_election.cc:334] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 2.160s (SENT)
W20260808 01:10:57.330752 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 84958.8us
I20260808 01:10:55.945820 1438 rpcz_store.cc:276] 0808 01:10:54.360806 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:54.426013 (+ 65207us) service_pool.cc:227] Handling call
0808 01:10:54.457692 (+ 31679us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:54.458009 (+ 317us) raft_consensus.cc:1520] Early marking committed up to index 4
0808 01:10:55.087092 (+629083us) log.cc:842] Serialized 14 byte log entry
0808 01:10:55.482034 (+394942us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:55.482890 (+ 856us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:10:55.485879 (+ 2989us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:55.493786 (+ 7907us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:55.518122 (+ 24336us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1917,"prepare.queue_time_us":638293,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":45189,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":72,"thread_start_us":987049,"threads_started":3,"wal-append.queue_time_us":338218,"wal-append.run_cpu_time_us":1773,"wal-append.run_wall_time_us":1007749}
I20260808 01:10:56.859552 3861 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:57.244532 1597 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
W20260808 01:10:56.915596 3818 consensus_meta.cc:287] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.879s user 0.008s sys 0.008s
I20260808 01:10:57.323324 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 709) took 1479 ms. Trace:
I20260808 01:10:56.987375 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:57.319597 2202 raft_consensus.cc:2362] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3. Re-sending same reply.
W20260808 01:10:57.075481 3979 consensus_meta.cc:287] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 1.270s user 0.010s sys 0.007s
I20260808 01:10:57.245913 1442 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 8.
W20260808 01:10:57.241030 3929 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.068s user 0.016s sys 0.032s
I20260808 01:10:57.240046 3958 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:57.329953 1599 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:56.858693 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:57.085505 4045 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:57.236212 3940 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:57.085687 3933 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:57.387501 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:57.429425 1446 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:57.092677 4040 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:57.304450 3992 raft_consensus.cc:729] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:57.327065 3963 raft_consensus.cc:2781] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:10:57.323719 3742 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:57.371385 3929 raft_consensus.cc:686] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:56.899922 3990 consensus_queue.cc:260] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:56.860124 3859 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:56.876573 3987 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:56.879002 3828 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:57.370697 4042 raft_consensus.cc:2781] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:10:57.375842 3979 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:57.361202 4033 consensus_queue.cc:1035] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260808 01:10:57.371950 3818 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:10:57.371480 3931 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.037s sys 0.089s
I20260808 01:10:57.377444 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:57.380301 3840 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:57.383780 4045 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:57.425211 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:57.392431 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:57.396644 1444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1972) took 1992 ms. Trace:
I20260808 01:10:57.406574 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:57.556679 1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2443) took 1495 ms. Trace:
I20260808 01:10:56.850270 4031 consensus_queue.cc:1046] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.735s
W20260808 01:10:57.373579 1338 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.009s sys 0.028s
I20260808 01:10:57.407912 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:10:57.755441 1338 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.935s user 0.018s sys 0.030s
I20260808 01:10:57.750665 1444 rpcz_store.cc:276] 0808 01:10:55.404394 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:55.420133 (+ 15739us) service_pool.cc:227] Handling call
0808 01:10:55.440602 (+ 20469us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:55.489623 (+ 49021us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:55.489650 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:55.489692 (+ 42us) raft_consensus.cc:1631] Marking committed up to 2
0808 01:10:55.489727 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:57.396414 (+1906687us) spinlock_profiling.cc:243] Waited 48.8 ms on lock 0x7b540009db54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:57.396463 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:57.396583 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":112227584}
I20260808 01:10:57.637009 3911 raft_consensus.cc:3032] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:57.750239 1603 rpcz_store.cc:276] 0808 01:10:56.058874 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:56.059672 (+ 798us) service_pool.cc:227] Handling call
0808 01:10:56.060055 (+ 383us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:56.060319 (+ 264us) raft_consensus.cc:1520] Early marking committed up to index 5
0808 01:10:57.243514 (+1183195us) log.cc:842] Serialized 14 byte log entry
0808 01:10:57.553461 (+309947us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:57.553540 (+ 79us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:10:57.553620 (+ 80us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:57.553720 (+ 100us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:57.553911 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":793139,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":8450,"raft.queue_time_us":126825,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":25977,"thread_start_us":1434717,"threads_started":2}
I20260808 01:10:57.407229 1519 raft_consensus.cc:1261] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:10:57.412613 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:57.760111 1524 raft_consensus.cc:2379] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 3: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:10:57.413681 1516 raft_consensus.cc:1261] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:10:57.415158 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:57.761333 1517 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 8.
I20260808 01:10:57.572431 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:57.387351 1445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1965) took 2268 ms. Trace:
I20260808 01:10:57.397176 4036 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:57.594731 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:57.785037 1446 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:10:57.752103 3929 consensus_queue.cc:237] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:10:57.373503 1410 connection.cc:232] RPC call timeout handler was delayed by -2.1071580449934117s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1865s for a response
W20260808 01:10:57.632433 4028 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.008s sys 0.030s
I20260808 01:10:57.393381 1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2446) took 1322 ms. Trace:
W20260808 01:10:57.787798 1410 connection.cc:232] RPC call timeout handler was delayed by -2.10175804499886s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18710000000000002s for a response
W20260808 01:10:57.793630 1410 connection.cc:232] RPC call timeout handler was delayed by -2.0315580450114794s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19490000000000002s for a response
I20260808 01:10:57.374792 3958 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:57.752126 4042 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:10:57.542563 1487 leader_election.cc:334] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.547s (SENT)
W20260808 01:10:57.795630 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 253075us
W20260808 01:10:57.796366 1487 leader_election.cc:334] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.580s (SENT)
W20260808 01:10:57.797469 1487 leader_election.cc:334] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.631s (SENT)
I20260808 01:10:57.797978 1487 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:57.552650 3933 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:57.755779 1439 raft_consensus.cc:2379] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 3: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:57.803750 1443 raft_consensus.cc:1226] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:57.803968 1439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55042 (request call id 2262) took 1815 ms. Trace:
I20260808 01:10:57.386093 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:57.407459 3916 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:57.388890 1489 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:57.807528 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:10:57.793284 4028 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.719s user 0.009s sys 0.033s
I20260808 01:10:57.763155 1445 rpcz_store.cc:276] 0808 01:10:55.118985 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:55.120508 (+ 1523us) service_pool.cc:227] Handling call
0808 01:10:55.130668 (+ 10160us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:56.719582 (+1588914us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:57.386945 (+667363us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b540007e254. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:57.387220 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3652621568}
I20260808 01:10:57.781391 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:57.411917 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:10:57.755488 3931 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.913s user 0.040s sys 0.095s
I20260808 01:10:57.398984 4043 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:10:57.387599 3809 consensus_meta.cc:287] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.672s user 0.006s sys 0.002s
I20260808 01:10:57.794653 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
W20260808 01:10:57.794325 1410 connection.cc:232] RPC call timeout handler was delayed by -1.8848580449994188s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21120000000000003s for a response
I20260808 01:10:57.748658 3963 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:10:57.411526 3954 consensus_meta.cc:287] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.402s user 0.012s sys 0.000s
W20260808 01:10:57.799780 1487 leader_election.cc:334] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.862s (SENT)
I20260808 01:10:57.819223 1487 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:57.820358 2209 raft_consensus.cc:2396] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:10:57.755616 1525 raft_consensus.cc:3037] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:57.803958 4036 raft_consensus.cc:2726] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:10:57.793558 1596 rpcz_store.cc:276] 0808 01:10:56.070750 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:56.070977 (+ 227us) service_pool.cc:227] Handling call
0808 01:10:56.071378 (+ 401us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:10:56.071821 (+ 443us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:56.071910 (+ 89us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:10:56.072442 (+ 532us) log.cc:842] Serialized 26 byte log entry
0808 01:10:56.383467 (+311025us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:10:56.383526 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:56.383629 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:10:57.393180 (+1009551us) raft_consensus.cc:1671] finished
0808 01:10:57.393201 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:57.393322 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1668,"raft.queue_time_us":921,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":32802,"thread_start_us":216622,"threads_started":1,"wal-append.queue_time_us":643325}
I20260808 01:10:57.779584 3818 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:57.825376 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:57.802779 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:10:57.805078 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 473544us
I20260808 01:10:57.805528 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:57.830061 1444 raft_consensus.cc:2396] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:10:57.405094 3996 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:57.552999 4034 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:57.569732 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:57.804437 1439 rpcz_store.cc:276] 0808 01:10:55.988392 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:57.328628 (+1340236us) service_pool.cc:227] Handling call
0808 01:10:57.803882 (+475254us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:57.807528 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:57.844552 1446 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:10:57.397011 3795 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:57.770403 3911 raft_consensus.cc:729] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:57.413618 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:57.404004 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:57.855732 2205 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:57.856791 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:57.857990 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:57.414356 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:57.858610 3783 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:57.858865 1520 raft_consensus.cc:3037] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:57.851425 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:57.861011 1602 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:10:57.809731 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:57.808979 1519 raft_consensus.cc:2454] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:57.810834 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:57.811209 1521 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:10:57.810632 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 421777us
I20260808 01:10:57.797811 3991 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:57.750797 4040 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:57.814767 1410 connection.cc:232] RPC call timeout handler was delayed by -1.3529580449976493s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2703s for a response
I20260808 01:10:57.815388 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:57.871038 1516 raft_consensus.cc:3037] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:57.797573 3873 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:10:57.807559 3689 consensus_meta.cc:287] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 1.233s user 0.008s sys 0.004s
I20260808 01:10:57.799928 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:57.802197 4042 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:57.827138 1517 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:10:57.809734 4031 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:57.809933 3990 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:57.831439 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:10:57.831935 1487 leader_election.cc:334] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.872s (SENT)
W20260808 01:10:57.890625 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 58717.3us
W20260808 01:10:57.891287 1487 leader_election.cc:334] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.930s (SENT)
I20260808 01:10:57.834760 1444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55042 (request call id 2274) took 1602 ms. Trace:
I20260808 01:10:57.814435 3943 raft_consensus.cc:2781] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:10:57.392630 3806 consensus_queue.cc:260] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:57.822185 3987 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:57.820704 3963 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:57.849943 3940 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:57.863957 1524 raft_consensus.cc:2379] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 3: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:10:57.864459 1489 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:57.865319 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:57.810197 3933 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:57.863512 1518 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:10:57.870941 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:57.871423 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:10:57.801002 3799 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.959s user 0.003s sys 0.012s
I20260808 01:10:57.854981 1523 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:57.871244 1525 raft_consensus.cc:2454] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:57.906741 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:57.907392 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:57.869838 3795 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:57.880087 3911 consensus_queue.cc:260] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:57.881474 3748 consensus_queue.cc:1046] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:10:57.882534 1439 raft_consensus.cc:1226] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:57.884653 1442 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1975) took 1905 ms. Trace:
I20260808 01:10:57.884749 4009 consensus_queue.cc:1046] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:10:57.885198 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:57.925459 1439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55042 (request call id 2261) took 1937 ms. Trace:
I20260808 01:10:57.869498 2205 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:57.886019 1410 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:57.893689 1444 rpcz_store.cc:276] 0808 01:10:56.232523 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:57.784138 (+1551615us) service_pool.cc:227] Handling call
0808 01:10:57.834693 (+ 50555us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:10:57.894558 1565 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.006s sys 0.028s
W20260808 01:10:57.929186 1565 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.607s user 0.007s sys 0.028s
I20260808 01:10:57.929070 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:10:57.929527 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 607776us
I20260808 01:10:57.895974 1520 raft_consensus.cc:2454] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:57.896361 1445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55042 (request call id 2280) took 1126 ms. Trace:
I20260808 01:10:57.930341 1516 raft_consensus.cc:2454] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:57.930645 1445 rpcz_store.cc:276] 0808 01:10:56.770098 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:57.817802 (+1047704us) service_pool.cc:227] Handling call
0808 01:10:57.896298 (+ 78496us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:57.826345 3954 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:57.852805 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:57.900234 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:57.904284 1603 raft_consensus.cc:1261] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:10:57.821683 3992 consensus_queue.cc:260] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:57.905704 1521 raft_consensus.cc:2454] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:57.943238 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:57.943252 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:57.944226 1519 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:57.906474 1596 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 3: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:10:57.850333 4034 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:57.822726 4043 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:57.911074 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 46765.3us
I20260808 01:10:57.895334 3943 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:57.926093 1439 rpcz_store.cc:276] 0808 01:10:55.988014 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:57.863822 (+1875808us) service_pool.cc:227] Handling call
0808 01:10:57.882331 (+ 18509us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:57.925127 (+ 42796us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:57.925400 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:57.925166 1442 rpcz_store.cc:276] 0808 01:10:55.978924 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:57.863022 (+1884098us) service_pool.cc:227] Handling call
0808 01:10:57.884171 (+ 21149us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:57.884348 (+ 177us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:57.884371 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:57.884408 (+ 37us) raft_consensus.cc:1631] Marking committed up to 2
0808 01:10:57.884442 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:57.884497 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:57.884598 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:57.940243 4009 consensus_queue.cc:1225] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer 6b09dbefd24b4d52b385c4dff5697dc5 log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260808 01:10:57.935271 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 49318.9us
W20260808 01:10:57.936244 2171 connection.cc:232] RPC call timeout handler was delayed by -2.5303554790152702s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1784s for a response
I20260808 01:10:57.966981 1442 raft_consensus.cc:1261] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260808 01:10:57.967238 2171 connection.cc:232] RPC call timeout handler was delayed by -2.3755554790259339s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1956s for a response
I20260808 01:10:57.967621 1442 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55042 (request call id 2264) took 1978 ms. Trace:
W20260808 01:10:57.967684 2171 connection.cc:232] RPC call timeout handler was delayed by -2.2261554790311493s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2122s for a response
W20260808 01:10:57.968394 2171 connection.cc:232] RPC call timeout handler was delayed by -2.1820554790319875s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2171s for a response
I20260808 01:10:57.930004 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:57.940435 2202 raft_consensus.cc:2454] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:57.969120 1519 raft_consensus.cc:3032] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:57.941121 3911 raft_consensus.cc:3037] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:57.969429 1524 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:57.969555 1519 raft_consensus.cc:729] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:57.941675 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:57.970710 1517 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:57.907189 3840 raft_consensus.cc:2726] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:57.947089 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:57.907321 3940 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:57.891111 3873 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:57.953992 1600 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:57.954775 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:57.955408 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:57.888211 3991 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:57.906471 3799 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.065s user 0.003s sys 0.012s
I20260808 01:10:57.897821 4031 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:57.967437 1410 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:57.945899 1520 raft_consensus.cc:1226] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:57.968065 1442 rpcz_store.cc:276] 0808 01:10:55.989044 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:57.966131 (+1977087us) service_pool.cc:227] Handling call
0808 01:10:57.966541 (+ 410us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:57.967367 (+ 826us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:57.967567 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:57.956915 3992 raft_consensus.cc:3037] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:57.969132 1446 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:57.974552 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:57.979456 2171 leader_election.cc:398] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for earlier term 5. Current term is 6.
I20260808 01:10:57.946331 3958 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:57.980326 1444 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:57.923457 3987 raft_consensus.cc:2781] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:57.981276 1521 raft_consensus.cc:1261] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:57.947995 4033 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:10:57.970737 1519 consensus_queue.cc:260] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:57.984440 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:57.986984 1596 raft_consensus.cc:2454] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:57.924170 3795 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:57.991365 1439 raft_consensus.cc:1261] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:10:57.991122 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:58.079290 1439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55042 (request call id 2263) took 2090 ms. Trace:
I20260808 01:10:57.960846 3943 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:57.956877 3859 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:58.089443 4031 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:58.090466 4040 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:58.092449 4062 log.cc:924] Time spent T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.169s user 0.000s sys 0.000s
I20260808 01:10:58.093384 3954 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:58.094741 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:58.062397 1518 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:58.062688 2171 leader_election.cc:401] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260808 01:10:58.063230 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:58.066262 1445 raft_consensus.cc:1226] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:58.098785 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:58.069311 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:58.072290 1519 raft_consensus.cc:3037] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:57.972266 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:58.100564 1597 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:57.973615 3742 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:10:58.072582 4055 log.cc:924] Time spent T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.182s user 0.000s sys 0.000s
I20260808 01:10:58.079986 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:58.071990 4033 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:57.979683 3828 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:58.084116 4034 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:58.086404 4043 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:58.087080 3933 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:58.088579 2201 raft_consensus.cc:2379] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 2: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:10:57.985690 3979 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:57.987564 4045 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:58.088922 1444 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
W20260808 01:10:58.073205 4063 log.cc:924] Time spent T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20260808 01:10:58.107858 4031 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:58.099779 1521 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:10:58.105895 2202 raft_consensus.cc:2396] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:10:58.106264 1410 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:58.106910 1445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 626) took 2114 ms. Trace:
I20260808 01:10:58.097049 1442 raft_consensus.cc:2932] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:10:58.081214 4059 log.cc:924] Time spent T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.145s user 0.000s sys 0.001s
I20260808 01:10:58.074205 4009 raft_consensus.cc:2726] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:10:58.068199 3987 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:58.143311 3987 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:58.116863 1599 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:10:58.147560 4068 log.cc:924] Time spent T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20260808 01:10:58.121496 1524 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:10:58.149225 1524 raft_consensus.cc:1226] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:58.116937 3931 consensus_queue.cc:1035] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260808 01:10:58.150770 2204 raft_consensus.cc:1226] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 1. Current term is 3. Ops: []
I20260808 01:10:58.120265 3799 consensus_queue.cc:1035] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260808 01:10:58.139780 3979 raft_consensus.cc:2781] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:58.153096 3979 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:58.153566 3979 raft_consensus.cc:3037] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:58.153841 3991 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:58.155159 1443 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 623) took 2682 ms. Trace:
I20260808 01:10:58.155424 3873 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:58.140173 4031 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:58.122224 3915 raft_consensus.cc:3032] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:10:58.156919 3915 raft_consensus.cc:729] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:58.157069 3940 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:58.123013 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:10:58.123824 4021 log.cc:924] Time spent T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 1.292s user 0.000s sys 0.022s
I20260808 01:10:58.123695 1489 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:58.124296 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:58.079123 3958 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:58.163542 1517 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:58.162091 1519 raft_consensus.cc:1261] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260808 01:10:58.130385 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:10:58.131251 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 162160us
I20260808 01:10:58.131808 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:58.183946 1597 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:58.133400 1445 rpcz_store.cc:276] 0808 01:10:55.992214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.065660 (+2073446us) service_pool.cc:227] Handling call
0808 01:10:58.066048 (+ 388us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:58.106554 (+ 40506us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:58.106832 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:58.133687 1410 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:58.124657 3689 raft_consensus.cc:2726] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for earlier term 5. Current term is 6.
I20260808 01:10:58.125380 4033 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:10:58.131978 1568 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:57.361225 2208 rpcz_store.cc:276] 0808 01:10:55.843664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:55.843991 (+ 327us) service_pool.cc:227] Handling call
0808 01:10:55.844379 (+ 388us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:55.844668 (+ 289us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 01:10:56.563039 (+718371us) log.cc:842] Serialized 14 byte log entry
0808 01:10:57.322837 (+759798us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:57.322902 (+ 65us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:10:57.322977 (+ 75us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:57.323076 (+ 99us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:57.323243 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":64670,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":1688,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":33853,"thread_start_us":1429978,"threads_started":2}
I20260808 01:10:58.199149 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:58.200456 2208 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:58.157763 4031 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:58.180976 3748 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:58.206796 3748 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:58.209210 3748 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:58.120752 3801 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:58.209918 3801 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:58.212081 3801 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:58.155553 1443 rpcz_store.cc:276] 0808 01:10:55.472331 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:55.472788 (+ 457us) service_pool.cc:227] Handling call
0808 01:10:55.489988 (+ 17200us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:57.807206 (+2317218us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:58.154922 (+347716us) spinlock_profiling.cc:243] Waited 2.31 s on lock 0x7b54001d1354. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:58.155096 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5314910592}
I20260808 01:10:58.213414 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:58.143719 3795 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:58.218356 4042 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:58.219893 1525 raft_consensus.cc:2379] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:10:58.188712 3958 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:58.206655 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:58.220791 1602 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:58.221860 3958 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:58.224931 3987 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:58.228015 1602 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:10:58.228892 1567 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:58.229717 3958 raft_consensus.cc:2781] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:58.230110 3958 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:58.230432 3958 raft_consensus.cc:3037] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:10:58.184765 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 61151.4us
I20260808 01:10:58.079710 1439 rpcz_store.cc:276] 0808 01:10:55.988729 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:57.975207 (+1986478us) service_pool.cc:227] Handling call
0808 01:10:57.990938 (+ 15731us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:58.078978 (+ 88040us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:58.079227 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:58.237740 1439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1978) took 1894 ms. Trace:
I20260808 01:10:58.186731 2171 leader_election.cc:398] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5.
I20260808 01:10:58.238797 2171 leader_election.cc:401] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260808 01:10:58.238467 3795 consensus_queue.cc:1035] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:58.239812 1489 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:58.238194 1439 rpcz_store.cc:276] 0808 01:10:56.343659 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.236703 (+1893044us) service_pool.cc:227] Handling call
0808 01:10:58.237148 (+ 445us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:58.237349 (+ 201us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:58.237381 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:58.237433 (+ 52us) raft_consensus.cc:1631] Marking committed up to 2
0808 01:10:58.237492 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:58.237555 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:58.237676 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:58.187467 1519 pending_rounds.cc:76] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5: Aborting all ops after (but not including) 4
I20260808 01:10:58.188146 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:58.132223 4040 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
W20260808 01:10:58.188956 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 55320us
I20260808 01:10:58.190176 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:58.185679 1520 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:10:58.249028 1410 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:58.249993 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:58.250798 3799 raft_consensus.cc:2726] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:58.196296 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:10:58.127710 1565 leader_election.cc:334] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.076s (SENT)
I20260808 01:10:58.251708 1596 raft_consensus.cc:2354] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4.
I20260808 01:10:58.251855 1565 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:58.252475 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:58.205658 3940 raft_consensus.cc:2781] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:10:58.188112 1517 raft_consensus.cc:1226] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
W20260808 01:10:58.253074 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 125391us
I20260808 01:10:58.243621 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:58.244330 1523 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:58.242841 1519 pending_rounds.cc:97] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260808 01:10:58.204448 4009 raft_consensus.cc:2781] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:58.245060 4067 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:58.245097 1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2436) took 3031 ms. Trace:
I20260808 01:10:58.237766 3958 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:58.245604 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:58.245882 3689 raft_consensus.cc:2726] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5.
W20260808 01:10:58.245604 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 59043.5us
I20260808 01:10:58.248185 4042 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:58.250238 1410 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:58.192822 1522 raft_consensus.cc:1226] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: []
I20260808 01:10:58.252548 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:58.277262 2171 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:58.277511 1410 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:58.192821 1599 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:58.279271 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:58.272801 3816 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:58.280525 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:58.280380 3816 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:58.281087 1599 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:58.282641 3816 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:58.283233 2205 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:58.283789 3911 raft_consensus.cc:2726] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:10:58.266381 4067 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:58.284751 3799 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:10:58.285144 3799 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:58.285544 3799 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:10:58.285650 4044 log.cc:924] Time spent T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 1.045s user 0.001s sys 0.001s
I20260808 01:10:58.286458 4067 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:10:58.205387 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 73529.3us
I20260808 01:10:58.288422 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:58.288582 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:58.289064 1596 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:58.290885 3963 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:58.254899 1519 raft_consensus.cc:2866] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:10:58.272980 3748 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:58.293107 3748 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:58.294008 1519 raft_consensus.cc:1261] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260808 01:10:58.267153 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:58.266580 1598 rpcz_store.cc:276] 0808 01:10:55.214014 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:55.456767 (+242753us) service_pool.cc:227] Handling call
0808 01:10:55.485696 (+ 28929us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:10:55.509125 (+ 23429us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:55.703673 (+194548us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:10:55.704364 (+ 691us) log.cc:842] Serialized 26 byte log entry
0808 01:10:56.100209 (+395845us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:10:56.100261 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:56.100359 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:10:58.123134 (+2022775us) spinlock_profiling.cc:243] Waited 19 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:10:58.241115 (+117981us) raft_consensus.cc:1671] finished
0808 01:10:58.241146 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:58.245006 (+ 3860us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":195968,"raft.queue_time_us":22958,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":83452,"spinlock_wait_cycles":43636608,"thread_start_us":394659,"threads_started":1,"wal-append.queue_time_us":1069650}
I20260808 01:10:58.204124 3873 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:58.203495 3970 raft_consensus.cc:2781] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:58.276495 3940 raft_consensus.cc:686] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:58.286924 3802 raft_consensus.cc:3032] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:58.296335 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:58.295644 1523 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:10:58.307487 1598 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:58.274000 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:58.249050 1594 raft_consensus.cc:2362] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5. Re-sending same reply.
I20260808 01:10:58.299222 2208 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:58.300468 1489 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:58.238523 1567 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:58.302980 1596 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:58.305667 1517 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:58.317262 1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2448) took 1463 ms. Trace:
I20260808 01:10:58.315157 3689 consensus_queue.cc:1046] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:10:58.273403 4040 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:58.266674 3929 raft_consensus.cc:2781] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:58.309260 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:10:58.327649 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 89215.7us
I20260808 01:10:58.327988 1519 raft_consensus.cc:1226] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 5. Current term is 6. Ops: []
I20260808 01:10:58.315055 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:58.312537 3802 raft_consensus.cc:729] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:58.329840 1517 raft_consensus.cc:1226] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:58.264700 4009 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:58.320676 2171 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:58.295625 3987 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:58.264058 3795 raft_consensus.cc:2726] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:58.321832 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 975) took 1091 ms. Trace:
I20260808 01:10:58.314144 3722 tablet_copy_client.cc:531] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:10:58.308259 1599 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:10:58.322489 1600 rpcz_store.cc:276] 0808 01:10:56.853602 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:56.853771 (+ 169us) service_pool.cc:227] Handling call
0808 01:10:58.317072 (+1463301us) spinlock_profiling.cc:243] Waited 710 ms on lock 0x7b54001ea354. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:58.317196 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1633024256}
I20260808 01:10:58.323617 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:58.325165 1489 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:58.189008 3915 consensus_queue.cc:260] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:58.327790 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:10:58.315743 3931 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:58.307098 3801 raft_consensus.cc:2781] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:58.272290 3943 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:58.308504 2205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 964) took 1300 ms. Trace:
I20260808 01:10:58.331753 1567 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:58.444470 1517 raft_consensus.cc:1226] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:58.445789 1519 raft_consensus.cc:1226] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260808 01:10:58.366948 1523 raft_consensus.cc:1226] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 5. Current term is 6. Ops: []
I20260808 01:10:58.402189 1598 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:58.483755 2208 rpcz_store.cc:276] 0808 01:10:57.230212 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.198800 (+968588us) service_pool.cc:227] Handling call
0808 01:10:58.321783 (+122983us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:58.377401 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:58.393882 1597 raft_consensus.cc:2354] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 4. Current term is 5.
I20260808 01:10:58.306875 3970 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:58.490053 3944 raft_consensus.cc:2781] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:58.520197 1594 raft_consensus.cc:1226] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
W20260808 01:10:58.529636 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 209052us
I20260808 01:10:58.519986 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:10:58.485666 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 154020us
I20260808 01:10:58.493157 1599 raft_consensus.cc:1226] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 01:10:58.492759 1596 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:10:58.330302 3929 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:58.522504 1519 raft_consensus.cc:1226] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20260808 01:10:58.521399 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 196275us
I20260808 01:10:58.368798 3979 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:58.317045 3873 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:58.526074 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:58.521478 1517 raft_consensus.cc:1226] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260808 01:10:58.531513 3795 consensus_queue.cc:1046] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.370s
W20260808 01:10:58.770787 2202 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 982) took 1407 ms (client timeout 1516 ms). Trace:
I20260808 01:10:58.714386 1447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1974) took 2747 ms. Trace:
W20260808 01:10:58.722044 2207 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 957) took 1918 ms (client timeout 1631 ms). Trace:
W20260808 01:10:58.620924 1604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2430) took 3651 ms (client timeout 2000 ms). Trace:
I20260808 01:10:58.758603 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:58.901427 1524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1839) took 1056 ms. Trace:
I20260808 01:10:58.760299 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:10:58.902190 2202 rpcz_store.cc:269] 0808 01:10:57.363185 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:57.969624 (+606439us) service_pool.cc:227] Handling call
0808 01:10:58.770569 (+800945us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b54006b03d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:58.770723 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38852992}
I20260808 01:10:58.681211 1442 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55042 (request call id 2265) took 2691 ms. Trace:
I20260808 01:10:58.329833 4042 raft_consensus.cc:2781] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 01:10:58.620916 2201 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 955) took 1834 ms (client timeout 1878 ms). Trace:
I20260808 01:10:58.446803 3748 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:58.919080 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 381379us
I20260808 01:10:58.331033 3816 raft_consensus.cc:2781] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:10:58.707813 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:58.922595 1519 raft_consensus.cc:1261] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:10:58.907464 1442 rpcz_store.cc:276] 0808 01:10:55.989373 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:57.992618 (+2003245us) service_pool.cc:227] Handling call
0808 01:10:58.096727 (+104109us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:58.096931 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 01:10:58.303117 (+206186us) log.cc:842] Serialized 14 byte log entry
0808 01:10:58.680667 (+377550us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:58.680752 (+ 85us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:10:58.680828 (+ 76us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:58.680927 (+ 99us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:58.681124 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13847,"raft.queue_time_us":94461,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":143,"thread_start_us":360928,"threads_started":1}
W20260808 01:10:58.923360 2206 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 947) took 2138 ms (client timeout 1991 ms). Trace:
I20260808 01:10:58.924043 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:58.490628 3931 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:58.924692 1442 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
W20260808 01:10:58.620971 4077 log.cc:924] Time spent T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23: Append to log took a long time: real 0.145s user 0.000s sys 0.001s
W20260808 01:10:58.904418 1604 rpcz_store.cc:269] 0808 01:10:54.969993 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:55.034429 (+ 64436us) service_pool.cc:227] Handling call
0808 01:10:58.620623 (+3586194us) spinlock_profiling.cc:243] Waited 2.18 s on lock 0x7b540018ced4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:58.620808 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5010835712}
W20260808 01:10:58.712736 2209 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 953) took 1926 ms (client timeout 1862 ms). Trace:
I20260808 01:10:58.902554 1447 rpcz_store.cc:276] 0808 01:10:55.966629 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:55.966811 (+ 182us) service_pool.cc:227] Handling call
0808 01:10:55.992318 (+ 25507us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:56.735687 (+743369us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:56.735729 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:56.735784 (+ 55us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:10:56.735857 (+ 73us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:58.714092 (+1978235us) spinlock_profiling.cc:243] Waited 743 ms on lock 0x7b5400210554. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:58.714162 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:58.714313 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1708985728}
I20260808 01:10:58.903823 2208 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
W20260808 01:10:58.912667 1487 leader_election.cc:334] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.131s (SENT)
I20260808 01:10:58.907979 1567 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:10:58.915015 2201 rpcz_store.cc:269] 0808 01:10:56.786747 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:57.757316 (+970569us) service_pool.cc:227] Handling call
0808 01:10:58.620689 (+863373us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b54007378d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:58.620838 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":629040384}
W20260808 01:10:58.903204 2207 rpcz_store.cc:269] 0808 01:10:56.803990 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.721789 (+1917799us) service_pool.cc:214] Skipping call since client already timed out
0808 01:10:58.721958 (+ 169us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 01:10:58.839561 4040 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:10:58.465404 4029 log.cc:924] Time spent T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 1.661s user 0.004s sys 0.021s
I20260808 01:10:58.904785 1524 rpcz_store.cc:276] 0808 01:10:57.845297 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.150033 (+304736us) service_pool.cc:227] Handling call
0808 01:10:58.150391 (+ 358us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:10:58.150822 (+ 431us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:58.150859 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:10:58.151368 (+ 509us) log.cc:842] Serialized 26 byte log entry
0808 01:10:58.303947 (+152579us) raft_consensus.cc:1631] Marking committed up to 2
0808 01:10:58.584646 (+280699us) log.cc:842] Serialized 14 byte log entry
0808 01:10:58.586401 (+ 1755us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:58.586501 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:10:58.901154 (+314653us) raft_consensus.cc:1671] finished
0808 01:10:58.901172 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:58.901346 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":136764,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":13857,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":20235,"thread_start_us":409170,"threads_started":2,"wal-append.queue_time_us":157620}
I20260808 01:10:58.471886 3963 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:10:58.772678 4033 consensus_meta.cc:287] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.540s user 0.009s sys 0.001s
I20260808 01:10:58.965656 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:58.967370 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:58.686511 4075 log.cc:924] Time spent T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260808 01:10:58.377130 3689 consensus_queue.cc:1225] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer a4862d0938c84fc4bd840e4c17acae78 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260808 01:10:58.545248 1490 leader_election.cc:334] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.979s (SENT)
W20260808 01:10:58.976635 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 431410us
I20260808 01:10:58.376909 4070 consensus_queue.cc:1046] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.551s
I20260808 01:10:58.925405 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:58.980077 2202 raft_consensus.cc:2362] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4. Re-sending same reply.
W20260808 01:10:58.980589 2202 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 960) took 1977 ms (client timeout 1930 ms). Trace:
W20260808 01:10:58.980943 2202 rpcz_store.cc:269] 0808 01:10:57.003079 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.907285 (+1904206us) service_pool.cc:227] Handling call
0808 01:10:58.980521 (+ 73236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:58.929714 3931 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:58.527084 3970 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:58.524752 3991 raft_consensus.cc:2741] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 01:10:58.926795 1568 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:58.713702 1522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2052) took 1305 ms. Trace:
I20260808 01:10:58.988113 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:58.996789 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 70067.1us
I20260808 01:10:58.585425 4036 raft_consensus.cc:2781] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:59.000391 1604 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:10:59.000816 4036 raft_consensus.cc:686] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:59.001034 1604 raft_consensus.cc:595] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:10:59.001379 1604 raft_consensus.cc:603] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:10:59.001389 1521 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260808 01:10:59.002986 1521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36560 (request call id 536) took 1047 ms. Trace:
I20260808 01:10:58.490528 4009 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:58.898452 4057 raft_consensus.cc:3032] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:58.537760 3990 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:10:58.959625 2208 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 956) took 2172 ms (client timeout 2131 ms). Trace:
W20260808 01:10:59.035738 2208 rpcz_store.cc:269] 0808 01:10:56.786981 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.525764 (+1738783us) service_pool.cc:227] Handling call
0808 01:10:58.959549 (+433785us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:10:59.037014 2208 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 971) took 1951 ms (client timeout 1935 ms). Trace:
W20260808 01:10:59.037482 2208 rpcz_store.cc:269] 0808 01:10:57.085218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.036756 (+1951538us) service_pool.cc:214] Skipping call since client already timed out
0808 01:10:59.036929 (+ 173us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 01:10:58.617398 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:10:58.960203 2201 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 958) took 2135 ms (client timeout 2117 ms). Trace:
I20260808 01:10:58.960510 1489 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:10:59.040366 2201 rpcz_store.cc:269] 0808 01:10:56.824919 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.960009 (+2135090us) service_pool.cc:214] Skipping call since client already timed out
0808 01:10:58.960139 (+ 130us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 01:10:58.600898 3944 raft_consensus.cc:686] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:59.044862 3944 consensus_queue.cc:237] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:58.961078 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 48443.2us
W20260808 01:10:59.047986 1487 leader_election.cc:334] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.178s (SENT)
I20260808 01:10:59.048557 1487 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:10:58.961231 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 53323.6us
W20260808 01:10:59.050117 3799 consensus_meta.cc:287] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.764s user 0.011s sys 0.002s
I20260808 01:10:59.050586 1567 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:59.051998 1567 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:58.721607 3943 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:59.053365 3943 raft_consensus.cc:2726] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:58.659734 3958 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:59.011444 4057 raft_consensus.cc:729] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:59.060500 3958 raft_consensus.cc:2781] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:59.061308 3958 raft_consensus.cc:686] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:59.061483 4057 consensus_queue.cc:260] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:58.916885 3929 raft_consensus.cc:3037] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:58.836557 3801 raft_consensus.cc:686] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:59.064473 3801 consensus_queue.cc:237] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:59.069911 3783 raft_consensus.cc:2781] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:59.070564 3783 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:59.071084 3783 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:59.072480 4045 raft_consensus.cc:2781] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:10:59.073058 4045 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:59.073288 3979 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:59.073585 4045 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:59.003598 4067 raft_consensus.cc:2781] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:10:59.074355 4067 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:10:59.074846 4067 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:59.062711 3958 consensus_queue.cc:237] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:59.050590 3799 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:59.086530 3799 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:10:59.096700 4079 log.cc:924] Time spent T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20260808 01:10:59.097620 4072 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:59.098507 4072 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:59.099047 3990 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:59.101889 4072 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:59.102051 3990 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:58.967968 4033 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:59.104948 4033 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 9 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:59.106313 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
W20260808 01:10:58.958850 2209 rpcz_store.cc:269] 0808 01:10:56.786262 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:57.749806 (+963544us) service_pool.cc:227] Handling call
0808 01:10:58.712384 (+962578us) spinlock_profiling.cc:243] Waited 9.2 ms on lock 0x7b5400446454. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:58.712625 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21166848}
I20260808 01:10:58.963874 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:58.918493 4042 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:58.916252 3915 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:10:58.522058 2205 rpcz_store.cc:276] 0808 01:10:57.004831 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:57.857740 (+852909us) service_pool.cc:227] Handling call
0808 01:10:58.305292 (+447552us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:58.969048 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.120437 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:58.536803 4031 consensus_queue.cc:1046] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.426s
I20260808 01:10:58.972330 3772 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:58.965713 3795 raft_consensus.cc:2781] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:10:58.962031 3816 raft_consensus.cc:686] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:58.930806 3954 raft_consensus.cc:3032] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:58.974172 2200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 701) took 3499 ms. Trace:
I20260808 01:10:58.329967 3911 consensus_queue.cc:1046] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.596s
I20260808 01:10:58.924503 2171 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
W20260808 01:10:58.923772 2206 rpcz_store.cc:269] 0808 01:10:56.784586 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.707403 (+1922817us) service_pool.cc:227] Handling call
0808 01:10:58.923286 (+215883us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:58.463660 3940 consensus_queue.cc:237] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:10:58.988405 1522 rpcz_store.cc:276] 0808 01:10:57.408197 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:57.408309 (+ 112us) service_pool.cc:227] Handling call
0808 01:10:57.408581 (+ 272us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:58.276963 (+868382us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:58.713375 (+436412us) spinlock_profiling.cc:243] Waited 784 ms on lock 0x7b540012e654. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:58.713623 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1803070848}
I20260808 01:10:59.001808 4036 consensus_queue.cc:237] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:59.003358 1521 rpcz_store.cc:276] 0808 01:10:57.955974 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.295570 (+339596us) service_pool.cc:227] Handling call
0808 01:10:59.000775 (+705205us) spinlock_profiling.cc:243] Waited 5.3 ms on lock 0x7b5800035d98. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adf851 00007fe349adf702 00007fe3529cfb09 00007fe35190b730 00007fe3519d179b 00007fe3519d1821 00007fe3519952b0 00007fe3519807f9 00007fe3519802a2 00007fe34b066684
0808 01:10:59.001004 (+ 229us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:59.002667 (+ 1663us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:59.002922 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12184832}
I20260808 01:10:58.719085 4014 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:59.139140 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:59.004949 2202 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 976) took 1774 ms (client timeout 1775 ms). Trace:
W20260808 01:10:59.140331 2202 rpcz_store.cc:269] 0808 01:10:57.230573 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.981591 (+1751018us) service_pool.cc:227] Handling call
0808 01:10:59.004893 (+ 23302us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:58.917958 3742 consensus_queue.cc:1035] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.070s
I20260808 01:10:59.006174 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.141952 1440 raft_consensus.cc:2379] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 3: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:59.042666 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:59.041045 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 117851us
I20260808 01:10:59.063305 4057 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:59.131496 3795 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:59.109587 1444 raft_consensus.cc:1226] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 8. Current term is 9. Ops: []
I20260808 01:10:58.968039 2207 raft_consensus.cc:2454] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:10:59.122833 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:10:59.148844 2208 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 974) took 1919 ms (client timeout 1954 ms). Trace:
W20260808 01:10:59.124958 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 151486us
I20260808 01:10:59.149412 1447 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
W20260808 01:10:59.149346 2208 rpcz_store.cc:269] 0808 01:10:57.229855 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.042276 (+1812421us) service_pool.cc:227] Handling call
0808 01:10:59.148768 (+106492us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.150545 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.150816 1568 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:59.124269 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:59.127246 1445 raft_consensus.cc:2379] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 4: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:59.151980 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:10:59.152695 2201 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 721) took 1909 ms (client timeout 1926 ms). Trace:
I20260808 01:10:59.150692 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:59.153141 2201 rpcz_store.cc:269] 0808 01:10:57.243642 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.041317 (+1797675us) service_pool.cc:227] Handling call
0808 01:10:59.152636 (+111319us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:10:59.153823 2208 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 979) took 1922 ms (client timeout 2116 ms). Trace:
W20260808 01:10:59.156098 2208 rpcz_store.cc:269] 0808 01:10:57.231569 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.150371 (+1918802us) service_pool.cc:227] Handling call
0808 01:10:59.153749 (+ 3378us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.130992 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.120692 4042 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:10:59.137888 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 213457us
I20260808 01:10:59.162665 1447 raft_consensus.cc:2379] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 8: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:59.105841 3991 consensus_queue.cc:1046] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260808 01:10:59.132543 2200 rpcz_store.cc:276] 0808 01:10:55.475200 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:55.475328 (+ 128us) service_pool.cc:227] Handling call
0808 01:10:55.496582 (+ 21254us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:10:55.745517 (+248935us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:55.745556 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:10:55.804899 (+ 59343us) log.cc:842] Serialized 26 byte log entry
0808 01:10:56.580949 (+776050us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:10:56.580997 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:56.581082 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:10:58.329457 (+1748375us) spinlock_profiling.cc:243] Waited 4.59 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:10:58.973767 (+644310us) raft_consensus.cc:1671] finished
0808 01:10:58.973796 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:58.974104 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":236402,"raft.queue_time_us":365418,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":12816,"spinlock_wait_cycles":10561280,"thread_start_us":775468,"threads_started":1,"wal-append.queue_time_us":917778}
I20260808 01:10:59.108879 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:10:59.173003 1448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 624) took 3695 ms. Trace:
I20260808 01:10:59.173550 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:10:59.173470 1448 rpcz_store.cc:276] 0808 01:10:55.477878 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:55.488807 (+ 10929us) service_pool.cc:227] Handling call
0808 01:10:55.710640 (+221833us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:57.329503 (+1618863us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:59.172773 (+1843270us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b540026bbd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:59.172949 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2736274176}
I20260808 01:10:59.174631 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:59.139904 2206 raft_consensus.cc:2362] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3. Re-sending same reply.
W20260808 01:10:59.176153 2206 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 986) took 1438 ms (client timeout 1547 ms). Trace:
W20260808 01:10:59.176554 2206 rpcz_store.cc:269] 0808 01:10:57.738375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.137389 (+1399014us) service_pool.cc:227] Handling call
0808 01:10:59.176093 (+ 38704us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.142848 1489 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:59.144744 3689 raft_consensus.cc:2781] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:10:59.186415 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.144415 3816 consensus_queue.cc:237] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:59.149010 2207 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 952) took 2363 ms (client timeout 2178 ms). Trace:
I20260808 01:10:59.126636 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:59.199257 2207 rpcz_store.cc:269] 0808 01:10:56.786025 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.960170 (+2174145us) service_pool.cc:227] Handling call
0808 01:10:59.148928 (+188758us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.200237 2209 raft_consensus.cc:2396] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260808 01:10:59.129338 1490 leader_election.cc:334] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.516s (SENT)
I20260808 01:10:59.200510 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:59.201094 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 71777.3us
I20260808 01:10:59.201723 2207 raft_consensus.cc:2396] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:10:59.129190 1439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1980) took 2441 ms. Trace:
I20260808 01:10:59.132419 3954 raft_consensus.cc:729] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:10:59.164477 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:59.127038 3818 consensus_queue.cc:1035] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20260808 01:10:59.188546 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:10:59.186843 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 44120.2us
I20260808 01:10:59.194654 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:10:59.198266 2202 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 984) took 1558 ms (client timeout 1688 ms). Trace:
W20260808 01:10:59.205914 2202 rpcz_store.cc:269] 0808 01:10:57.640041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.142551 (+1502510us) service_pool.cc:227] Handling call
0808 01:10:59.198170 (+ 55619us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.206053 2208 raft_consensus.cc:2362] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3. Re-sending same reply.
W20260808 01:10:59.207247 2208 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 997) took 1427 ms (client timeout 1682 ms). Trace:
W20260808 01:10:59.207818 2208 rpcz_store.cc:269] 0808 01:10:57.779491 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.164190 (+1384699us) service_pool.cc:227] Handling call
0808 01:10:59.207166 (+ 42976us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.209172 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:59.211040 2209 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 981) took 1848 ms (client timeout 1862 ms). Trace:
I20260808 01:10:59.205441 1440 raft_consensus.cc:2454] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
W20260808 01:10:59.213063 2209 rpcz_store.cc:269] 0808 01:10:57.362708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.108339 (+1745631us) service_pool.cc:227] Handling call
0808 01:10:59.210964 (+102625us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.213779 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.211972 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:59.214495 2208 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 998) took 1434 ms (client timeout 1775 ms). Trace:
W20260808 01:10:59.214921 2208 rpcz_store.cc:269] 0808 01:10:57.779775 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.208796 (+1429021us) service_pool.cc:227] Handling call
0808 01:10:59.214429 (+ 5633us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.215785 1440 raft_consensus.cc:2379] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 3: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:59.142225 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:59.214767 2201 raft_consensus.cc:2454] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
W20260808 01:10:59.222762 2201 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 994) took 1444 ms (client timeout 1646 ms). Trace:
W20260808 01:10:59.223249 2201 rpcz_store.cc:269] 0808 01:10:57.778635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.163592 (+1384957us) service_pool.cc:227] Handling call
0808 01:10:59.222685 (+ 59093us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.211925 1489 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:59.225862 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:59.147570 3907 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:59.231016 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.171780 3748 raft_consensus.cc:2781] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:10:59.232014 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.209931 1439 rpcz_store.cc:276] 0808 01:10:56.688209 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.242765 (+1554556us) service_pool.cc:227] Handling call
0808 01:10:58.278538 (+ 35773us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:58.278815 (+ 277us) raft_consensus.cc:1520] Early marking committed up to index 7
0808 01:10:58.321280 (+ 42465us) log.cc:842] Serialized 14 byte log entry
0808 01:10:59.128819 (+807539us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:59.128880 (+ 61us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:10:59.128930 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:59.129006 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:59.129128 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":44356,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":37901,"raft.queue_time_us":54691,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":21798,"thread_start_us":776770,"threads_started":2,"wal-append.queue_time_us":644557}
I20260808 01:10:59.186527 3689 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:59.275630 1447 raft_consensus.cc:2454] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:59.233119 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:59.233098 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:10:59.245343 2208 raft_consensus.cc:2362] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4. Re-sending same reply.
W20260808 01:10:59.277402 2208 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 988) took 1520 ms (client timeout 1930 ms). Trace:
W20260808 01:10:59.277822 2208 rpcz_store.cc:269] 0808 01:10:57.757539 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.222864 (+1465325us) service_pool.cc:227] Handling call
0808 01:10:59.277322 (+ 54458us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.278857 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:59.235527 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.279405 3911 consensus_queue.cc:1046] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:10:59.219228 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.253986 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.281536 1445 raft_consensus.cc:2379] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 3: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:59.281859 1525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58280 (request call id 1842) took 1434 ms. Trace:
I20260808 01:10:59.282284 1525 rpcz_store.cc:276] 0808 01:10:57.847824 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:57.907136 (+ 59312us) service_pool.cc:227] Handling call
0808 01:10:59.281648 (+1374512us) spinlock_profiling.cc:243] Waited 312 ms on lock 0x7b54001a1b54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:59.281786 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":716669696}
I20260808 01:10:59.282785 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.190876 3873 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:10:59.262280 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
W20260808 01:10:59.291502 2209 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 995) took 1512 ms (client timeout 1880 ms). Trace:
W20260808 01:10:59.291996 2200 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 972) took 2063 ms (client timeout 2243 ms). Trace:
W20260808 01:10:59.292104 2209 rpcz_store.cc:269] 0808 01:10:57.778893 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.213876 (+1434983us) service_pool.cc:227] Handling call
0808 01:10:59.291425 (+ 77549us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18816}
W20260808 01:10:59.292613 2200 rpcz_store.cc:269] 0808 01:10:57.229097 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.173107 (+1944010us) service_pool.cc:227] Handling call
0808 01:10:59.291918 (+118811us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.233475 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:59.164278 3795 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:59.267093 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:59.301978 2201 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 992) took 1524 ms (client timeout 1935 ms). Trace:
W20260808 01:10:59.302464 2201 rpcz_store.cc:269] 0808 01:10:57.778054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.224159 (+1446105us) service_pool.cc:227] Handling call
0808 01:10:59.301902 (+ 77743us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:10:59.233758 2207 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 993) took 1455 ms (client timeout 1744 ms). Trace:
I20260808 01:10:59.212437 3943 raft_consensus.cc:2781] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:59.317881 1523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36560 (request call id 548) took 1250 ms. Trace:
I20260808 01:10:59.262145 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:59.279556 2208 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:59.326536 2202 raft_consensus.cc:2362] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4. Re-sending same reply.
I20260808 01:10:59.298890 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:10:59.327333 2202 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 987) took 1577 ms (client timeout 1872 ms). Trace:
I20260808 01:10:59.280694 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:10:59.327927 2202 rpcz_store.cc:269] 0808 01:10:57.749776 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.212256 (+1462480us) service_pool.cc:227] Handling call
0808 01:10:59.327260 (+115004us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.297678 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:59.320791 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:59.307055 4082 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:59.308437 1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2158) took 1119 ms. Trace:
I20260808 01:10:59.307595 3840 raft_consensus.cc:2781] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:10:59.309367 1444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1977) took 3138 ms. Trace:
W20260808 01:10:59.311357 1446 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55040 (request call id 1979) took 2949 ms (client timeout 3080 ms). Trace:
W20260808 01:10:59.303822 2207 rpcz_store.cc:269] 0808 01:10:57.778359 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.200185 (+1421826us) service_pool.cc:227] Handling call
0808 01:10:59.233676 (+ 33491us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.313824 1598 raft_consensus.cc:1261] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260808 01:10:59.326584 2208 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 983) took 1723 ms (client timeout 2131 ms). Trace:
I20260808 01:10:59.326925 1443 raft_consensus.cc:1226] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 1. Current term is 2. Ops: []
I20260808 01:10:59.327418 1604 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:59.328392 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.321484 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:59.211081 4031 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20260808 01:10:59.281014 3748 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:10:59.304926 2206 raft_consensus.cc:2396] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 01:10:59.266515 3783 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:59.341356 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.359185 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.261327 2205 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:10:59.280665 3911 raft_consensus.cc:3032] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Stepping down as leader of term 4
W20260808 01:10:59.368700 2208 rpcz_store.cc:269] 0808 01:10:57.603689 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.278556 (+1674867us) service_pool.cc:227] Handling call
0808 01:10:59.326521 (+ 47965us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.794878 1599 raft_consensus.cc:1261] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:10:59.798633 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:59.798753 1440 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:10:59.449344 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:10:59.772234 1410 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.022s sys 0.044s
I20260808 01:10:59.799877 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:10:59.800401 1410 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.669s user 0.025s sys 0.044s
I20260808 01:10:59.800086 1447 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:59.800813 1440 raft_consensus.cc:2379] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 3: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:59.791090 1604 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:10:59.288654 3689 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:10:59.359802 1523 rpcz_store.cc:276] 0808 01:10:58.067159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.518378 (+451219us) service_pool.cc:227] Handling call
0808 01:10:58.598369 (+ 79991us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:10:58.598665 (+ 296us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:58.598696 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:10:58.618918 (+ 20222us) log.cc:842] Serialized 26 byte log entry
0808 01:10:59.127592 (+508674us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:10:59.127630 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:59.127712 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:10:59.317585 (+189873us) raft_consensus.cc:1671] finished
0808 01:10:59.317621 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:59.317805 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":221925,"threads_started":1,"wal-append.queue_time_us":577814}
I20260808 01:10:59.369022 1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2171) took 1173 ms. Trace:
I20260808 01:10:59.799293 2208 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
W20260808 01:10:59.597003 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 394983us
I20260808 01:10:59.799399 1442 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:10:59.359524 2209 raft_consensus.cc:2396] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:10:59.359557 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:10:59.519496 1602 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.006s sys 0.043s
W20260808 01:10:59.519620 1565 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.016s sys 0.038s
I20260808 01:10:59.804044 1598 rpcz_store.cc:276] 0808 01:10:58.195390 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.601419 (+406029us) service_pool.cc:227] Handling call
0808 01:10:59.313264 (+711845us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b5800236498. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adfc6c 00007fe349adfbd6 000000000034aea9 00007fe35190ba30 00007fe3519d17b4 00007fe3519d1821 00007fe3519952b0 00007fe3519807f9 00007fe3519802a2 00007fe34b066684
0808 01:10:59.313433 (+ 169us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:59.368758 (+ 55325us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:59.368966 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":712145152}
I20260808 01:10:59.370079 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
W20260808 01:10:59.800813 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 669290us
I20260808 01:10:59.429592 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:59.361692 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 948) took 2576 ms. Trace:
I20260808 01:10:59.808475 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:59.372371 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.809402 1445 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:10:59.805656 1448 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:59.809702 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:59.359185 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:59.810628 2202 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:59.805244 1442 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55042 (request call id 2306) took 1286 ms. Trace:
W20260808 01:10:59.804893 1490 leader_election.cc:334] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.646s (SENT)
W20260808 01:10:59.811301 2202 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 989) took 2034 ms (client timeout 2244 ms). Trace:
W20260808 01:10:59.723466 1603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2160) took 1534 ms (client timeout 2028 ms). Trace:
W20260808 01:10:59.811744 2202 rpcz_store.cc:269] 0808 01:10:57.777051 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.328844 (+1551793us) service_pool.cc:227] Handling call
0808 01:10:59.811242 (+482398us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:10:59.812038 1603 rpcz_store.cc:269] 0808 01:10:58.189317 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.191940 (+ 2623us) service_pool.cc:227] Handling call
0808 01:10:59.723114 (+1531174us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b5400121954. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:59.723345 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3271483776}
W20260808 01:10:59.812716 1490 leader_election.cc:334] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.700s (SENT)
W20260808 01:10:59.610343 1487 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.015s sys 0.020s
I20260808 01:10:59.813148 1490 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:59.814551 1440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55042 (request call id 2292) took 1510 ms. Trace:
W20260808 01:10:59.813514 1487 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.764s user 0.022s sys 0.025s
W20260808 01:10:59.806485 1565 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.846s user 0.022s sys 0.063s
I20260808 01:10:59.482331 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.376778 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:59.816349 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 768352us
W20260808 01:10:59.805701 1602 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.873s user 0.012s sys 0.056s
I20260808 01:10:59.801046 1447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55042 (request call id 2287) took 1502 ms. Trace:
I20260808 01:10:59.574362 1518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1850) took 1372 ms. Trace:
I20260808 01:10:59.811324 1442 rpcz_store.cc:276] 0808 01:10:58.518954 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.328671 (+809717us) service_pool.cc:227] Handling call
0808 01:10:59.805187 (+476516us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:10:59.816159 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 884345us
I20260808 01:10:59.813064 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:59.801607 1444 rpcz_store.cc:276] 0808 01:10:56.170830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.221160 (+2050330us) service_pool.cc:227] Handling call
0808 01:10:58.221542 (+ 382us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:59.163977 (+942435us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:59.309123 (+145146us) spinlock_profiling.cc:243] Waited 888 ms on lock 0x7b54001d2254. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:59.309298 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2041705344}
I20260808 01:10:59.816370 1440 rpcz_store.cc:276] 0808 01:10:58.303768 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.799630 (+1495862us) service_pool.cc:227] Handling call
0808 01:10:59.814495 (+ 14865us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.817587 1518 rpcz_store.cc:276] 0808 01:10:58.201867 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.740534 (+538667us) service_pool.cc:227] Handling call
0808 01:10:58.770735 (+ 30201us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:58.770967 (+ 232us) raft_consensus.cc:1520] Early marking committed up to index 4
0808 01:10:58.771328 (+ 361us) log.cc:842] Serialized 14 byte log entry
0808 01:10:59.573890 (+802562us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:59.573967 (+ 77us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:10:59.574033 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:59.574125 (+ 92us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:59.574288 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":571,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":85301,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":123,"thread_start_us":802151,"threads_started":2,"wal-append.queue_time_us":457414}
W20260808 01:10:59.535357 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 483446us
I20260808 01:10:59.817157 1447 rpcz_store.cc:276] 0808 01:10:58.298669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.409415 (+1110746us) service_pool.cc:227] Handling call
0808 01:10:59.800991 (+391576us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.539963 1443 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55042 (request call id 2270) took 3423 ms. Trace:
I20260808 01:10:59.818972 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.819555 1443 rpcz_store.cc:276] 0808 01:10:56.116550 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.214646 (+2098096us) service_pool.cc:227] Handling call
0808 01:10:59.324035 (+1109389us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:59.375919 (+ 51884us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:59.538483 (+162564us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540014cdd0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:59.539865 (+ 1382us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2550480256}
I20260808 01:10:59.820115 1440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55040 (request call id 1996) took 1494 ms. Trace:
I20260808 01:10:59.820631 1440 rpcz_store.cc:276] 0808 01:10:58.325724 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.818731 (+1493007us) service_pool.cc:227] Handling call
0808 01:10:59.820049 (+ 1318us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.821617 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:59.822170 1440 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:59.802372 1597 rpcz_store.cc:276] 0808 01:10:58.188658 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.243400 (+ 54742us) service_pool.cc:227] Handling call
0808 01:10:59.308206 (+1064806us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b540028f5d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:59.308371 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":282158848}
I20260808 01:10:59.827430 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:59.726572 1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:45612 (request call id 761) took 1528 ms. Trace:
I20260808 01:10:59.828532 1596 rpcz_store.cc:276] 0808 01:10:58.197879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.920902 (+723023us) service_pool.cc:227] Handling call
0808 01:10:59.725428 (+804526us) spinlock_profiling.cc:243] Waited 752 ms on lock 0x7b5400121954. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4af0 00007fe34ca47f2a 00007fe351984c6e 00007fe34b07b807 00007fe34b07b792 00007fe34b07b6c1 00007fe34b07b63c 00007fe34b07a8f2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:59.726432 (+ 1004us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1729145088}
W20260808 01:10:59.829548 2209 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 996) took 2050 ms (client timeout 2029 ms). Trace:
W20260808 01:10:59.829878 2209 rpcz_store.cc:269] 0808 01:10:57.779208 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.293847 (+1514639us) service_pool.cc:227] Handling call
0808 01:10:59.829483 (+535636us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:10:59.830545 2200 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 962) took 2826 ms (client timeout 2772 ms). Trace:
W20260808 01:10:59.830787 1487 leader_election.cc:334] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.646s (SENT)
W20260808 01:10:59.830859 2200 rpcz_store.cc:269] 0808 01:10:57.004060 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.293557 (+2289497us) service_pool.cc:227] Handling call
0808 01:10:59.830494 (+536937us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.831348 1487 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:59.807667 2204 rpcz_store.cc:276] 0808 01:10:56.784868 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:56.784985 (+ 117us) service_pool.cc:227] Handling call
0808 01:10:56.804561 (+ 19576us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:58.151260 (+1346699us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:59.361460 (+1210200us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54004dd5d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:10:59.361631 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3095483776}
W20260808 01:10:59.369251 1446 rpcz_store.cc:269] 0808 01:10:56.362170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:57.845186 (+1483016us) service_pool.cc:227] Handling call
0808 01:10:59.311162 (+1465976us) spinlock_profiling.cc:243] Waited 64.7 ms on lock 0x7b540014cdd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:10:59.311283 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":148847872}
I20260808 01:10:59.816524 2201 raft_consensus.cc:2454] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
W20260808 01:10:59.832846 1487 leader_election.cc:334] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.935s (SENT)
W20260808 01:10:59.833117 2201 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 985) took 2154 ms (client timeout 2178 ms). Trace:
W20260808 01:10:59.833489 2201 rpcz_store.cc:269] 0808 01:10:57.679169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.303517 (+1624348us) service_pool.cc:227] Handling call
0808 01:10:59.833072 (+529555us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:10:59.835197 2206 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 991) took 2057 ms (client timeout 1724 ms). Trace:
I20260808 01:10:59.639202 3722 tablet_bootstrap.cc:492] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap starting.
W20260808 01:10:59.617013 3801 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.001s sys 0.015s
W20260808 01:10:59.692747 4094 log.cc:924] Time spent T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.224s user 0.001s sys 0.000s
I20260808 01:10:59.838581 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:59.839162 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:10:59.713107 3929 consensus_meta.cc:287] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.649s user 0.007s sys 0.003s
W20260808 01:10:59.835573 2206 rpcz_store.cc:269] 0808 01:10:57.777711 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.177454 (+1399743us) service_pool.cc:227] Handling call
0808 01:10:59.835128 (+657674us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:10:59.826848 4067 consensus_meta.cc:287] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.751s user 0.012s sys 0.001s
I20260808 01:10:59.836372 2202 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:10:59.799943 3911 raft_consensus.cc:729] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:59.365387 3990 raft_consensus.cc:3032] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:10:59.850112 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:59.861964 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:59.844656 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:59.864296 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:59.865038 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:10:59.866074 3801 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s user 0.007s sys 0.024s
I20260808 01:10:59.868506 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:59.826937 3963 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:59.869127 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.928050 1443 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55042 (request call id 2311) took 1050 ms. Trace:
I20260808 01:10:59.803753 3806 raft_consensus.cc:3037] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:59.804450 3748 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:10:59.870759 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:10:59.542040 4086 raft_consensus.cc:3032] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:10:59.872465 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:59.841138 1410 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:10:59.811525 4014 consensus_meta.cc:287] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.670s user 0.006s sys 0.006s
I20260808 01:10:59.362658 4084 raft_consensus.cc:3032] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:10:59.847047 1568 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:10:59.328871 3970 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:59.860975 3931 consensus_queue.cc:1035] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260808 01:10:59.936712 3801 raft_consensus.cc:2781] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:10:59.937285 3801 raft_consensus.cc:686] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:10:59.884723 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:59.888237 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:10:59.937945 2204 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:10:59.383330 3943 raft_consensus.cc:686] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:10:59.938171 1444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55042 (request call id 2308) took 1301 ms. Trace:
I20260808 01:10:59.938090 3801 consensus_queue.cc:237] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:59.938645 1444 rpcz_store.cc:276] 0808 01:10:58.636588 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.836813 (+1200225us) service_pool.cc:227] Handling call
0808 01:10:59.938109 (+101296us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.935104 3940 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:59.940224 1444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1985) took 1875 ms. Trace:
I20260808 01:10:59.940346 3940 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:59.940577 1444 rpcz_store.cc:276] 0808 01:10:58.064980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.939324 (+1874344us) service_pool.cc:227] Handling call
0808 01:10:59.939657 (+ 333us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:59.939815 (+ 158us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:59.939932 (+ 117us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:59.939974 (+ 42us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:10:59.940010 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:59.940077 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:59.940176 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:10:59.890450 4057 consensus_meta.cc:287] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.746s user 0.011s sys 0.000s
I20260808 01:10:59.890712 4031 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:59.307118 3802 consensus_queue.cc:260] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:59.892496 4042 consensus_meta.cc:287] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.701s user 0.008s sys 0.003s
I20260808 01:10:59.893639 3929 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:59.894229 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:10:59.954288 1440 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:10:59.894905 3783 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:10:59.838898 3979 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:10:59.834487 3818 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:10:59.900645 4009 consensus_meta.cc:287] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.819s user 0.012s sys 0.000s
I20260808 01:10:59.901051 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:59.918473 4067 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:59.869913 3861 consensus_queue.cc:1035] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260808 01:10:59.902312 4101 raft_consensus.cc:2726] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:10:59.902989 4099 raft_consensus.cc:2726] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:10:59.904152 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:59.907709 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:10:59.753758 3915 consensus_meta.cc:287] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.633s user 0.010s sys 0.001s
I20260808 01:10:59.914746 1448 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:10:59.930943 3963 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:59.923789 1446 raft_consensus.cc:2379] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 6: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:10:59.868600 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:10:59.976636 1519 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:10:59.976900 1596 raft_consensus.cc:1226] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
I20260808 01:10:59.922549 1442 raft_consensus.cc:2379] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 3: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:10:59.922863 2201 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:10:59.956269 3802 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:10:59.923126 2209 raft_consensus.cc:2354] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4.
I20260808 01:10:59.923409 1522 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:10:59.924952 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:59.979013 1442 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55042 (request call id 2290) took 1679 ms. Trace:
I20260808 01:10:59.922026 3990 raft_consensus.cc:729] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:59.903570 4102 consensus_queue.cc:1035] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:10:59.834950 3999 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:59.987185 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:59.926702 1487 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:10:59.928354 1439 raft_consensus.cc:1226] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 7. Current term is 8. Ops: []
I20260808 01:10:59.928385 1443 rpcz_store.cc:276] 0808 01:10:58.878125 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.820593 (+942468us) service_pool.cc:227] Handling call
0808 01:10:59.927980 (+107387us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:10:59.834681 2205 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 973) took 2598 ms (client timeout 2000 ms). Trace:
I20260808 01:10:59.840080 3944 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:10:59.359728 4082 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:00.003402 1446 raft_consensus.cc:1226] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: []
W20260808 01:10:59.934413 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 93377.4us
I20260808 01:11:00.007381 1410 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:10:59.328333 3954 consensus_queue.cc:260] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:10:59.939051 3943 consensus_queue.cc:237] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:00.014770 1446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1995) took 1705 ms. Trace:
I20260808 01:11:00.015188 1446 rpcz_store.cc:276] 0808 01:10:58.308991 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.976301 (+1667310us) service_pool.cc:227] Handling call
0808 01:11:00.003164 (+ 26863us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:00.014470 (+ 11306us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:00.014703 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.979012 3799 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:59.934165 3865 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:59.954734 1445 raft_consensus.cc:1226] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:59.932180 4086 raft_consensus.cc:729] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:59.957696 1520 raft_consensus.cc:2379] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 3: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:10:59.958539 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:10:59.935123 4065 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:59.933423 4084 raft_consensus.cc:729] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:10:59.960235 1524 raft_consensus.cc:3037] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:10:59.865527 4045 consensus_meta.cc:287] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.791s user 0.014s sys 0.000s
I20260808 01:10:59.941574 3958 raft_consensus.cc:2741] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260808 01:10:59.961174 1440 raft_consensus.cc:1226] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 01:10:59.965492 1444 raft_consensus.cc:1226] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
I20260808 01:10:59.968806 1525 raft_consensus.cc:2379] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 3: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:10:59.945780 3992 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:59.971575 1448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1991) took 1676 ms. Trace:
I20260808 01:11:00.050539 1448 rpcz_store.cc:276] 0808 01:10:58.295058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.970784 (+1675726us) service_pool.cc:227] Handling call
0808 01:10:59.971097 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:59.971261 (+ 164us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:10:59.971284 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:10:59.971322 (+ 38us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:10:59.971355 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:59.971409 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:10:59.971515 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.910401 2207 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:10:59.956218 3783 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:10:59.972373 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:00.051805 2207 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 990) took 2274 ms (client timeout 2772 ms). Trace:
I20260808 01:10:59.972955 3970 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:10:59.973480 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:00.052412 2207 rpcz_store.cc:269] 0808 01:10:57.777407 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.369831 (+1592424us) service_pool.cc:227] Handling call
0808 01:11:00.051747 (+681916us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.979532 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:59.979718 1442 rpcz_store.cc:276] 0808 01:10:58.299384 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.818105 (+1518721us) service_pool.cc:227] Handling call
0808 01:10:59.978951 (+160846us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:10:59.988768 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 62153.8us
I20260808 01:11:00.053663 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
W20260808 01:11:00.055218 1487 leader_election.cc:334] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.516s (SENT)
I20260808 01:11:00.055781 1487 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:10:59.989121 1439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 627) took 2744 ms. Trace:
I20260808 01:11:00.057519 1439 rpcz_store.cc:276] 0808 01:10:57.245014 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.860106 (+2615092us) service_pool.cc:227] Handling call
0808 01:10:59.928176 (+ 68070us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:59.988873 (+ 60697us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:10:59.989054 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.956396 4036 consensus_queue.cc:1035] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260808 01:10:59.990332 1519 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 8.
W20260808 01:10:59.996030 1565 leader_election.cc:334] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.568s (SENT)
I20260808 01:11:00.008664 1410 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
W20260808 01:11:00.060331 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 51744.4us
I20260808 01:10:59.953860 4031 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:59.948490 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
W20260808 01:11:00.060660 1487 leader_election.cc:334] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.688s (SENT)
I20260808 01:11:00.067293 1487 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:00.052088 2206 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:00.068442 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:00.014591 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:59.957583 4042 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:10:59.991315 2205 rpcz_store.cc:269] 0808 01:10:57.229475 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.131724 (+1902249us) service_pool.cc:227] Handling call
0808 01:10:59.828014 (+696290us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:00.098706 1601 raft_consensus.cc:2396] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:11:00.032976 3990 consensus_queue.cc:260] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:59.987152 1438 raft_consensus.cc:1261] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20260808 01:10:59.961571 3979 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:00.025091 2201 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:11:00.110179 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:00.065871 3801 raft_consensus.cc:2781] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:00.067013 3861 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:00.123978 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:00.125088 1603 raft_consensus.cc:2379] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 3: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:00.039027 2204 raft_consensus.cc:2454] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:11:00.042641 1445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 628) took 2192 ms. Trace:
I20260808 01:11:00.128032 1445 rpcz_store.cc:276] 0808 01:10:57.850497 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.935687 (+2085190us) service_pool.cc:227] Handling call
0808 01:10:59.954571 (+ 18884us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:00.042291 (+ 87720us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:00.042564 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:10:59.953603 3911 consensus_queue.cc:260] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:00.033913 4082 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:00.047326 1440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1990) took 1752 ms. Trace:
I20260808 01:11:00.131019 1440 rpcz_store.cc:276] 0808 01:10:58.294759 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.955041 (+1660282us) service_pool.cc:227] Handling call
0808 01:10:59.955364 (+ 323us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:00.047062 (+ 91698us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:00.047268 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:00.047339 1444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1988) took 1753 ms. Trace:
I20260808 01:11:00.132193 1444 rpcz_store.cc:276] 0808 01:10:58.294198 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.945943 (+1651745us) service_pool.cc:227] Handling call
0808 01:10:59.962015 (+ 16072us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:00.047158 (+ 85143us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:00.047293 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:00.049563 1446 raft_consensus.cc:1226] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:00.059399 1565 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:00.052942 2202 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:11:00.070101 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:00.070897 2209 raft_consensus.cc:2379] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 3: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:00.071733 1410 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:00.073845 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:00.076459 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:00.135965 1597 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:11:00.096242 1487 leader_election.cc:334] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.954s (SENT)
I20260808 01:11:00.077525 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:00.136821 1487 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:00.039405 3799 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:00.041643 4015 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:10:59.991500 3999 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:00.111195 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:10:59.962358 3818 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:10:59.965989 4009 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:00.051132 3940 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:00.066140 4101 raft_consensus.cc:2726] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 01:11:00.111938 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 275053us
I20260808 01:11:00.070562 3931 raft_consensus.cc:2781] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:11:00.115511 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:00.165215 2206 raft_consensus.cc:2362] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3. Re-sending same reply.
I20260808 01:11:00.110874 2200 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
W20260808 01:11:00.121577 3795 consensus_meta.cc:287] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.592s user 0.009s sys 0.001s
I20260808 01:11:00.121912 3970 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:00.122506 3990 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:00.129148 1439 raft_consensus.cc:1226] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:00.107038 1438 pending_rounds.cc:76] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d: Aborting all ops after (but not including) 4
I20260808 01:11:00.132985 1599 raft_consensus.cc:1226] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:00.135784 1594 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:00.038205 3944 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:00.137210 1598 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:00.145733 4121 raft_consensus.cc:2726] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:00.095506 4057 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:00.138024 1446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 633) took 1296 ms. Trace:
I20260808 01:11:00.103622 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:00.127281 1524 raft_consensus.cc:2454] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:11:00.134245 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:00.139583 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 143574us
I20260808 01:11:00.210659 1446 rpcz_store.cc:276] 0808 01:10:58.842015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.016275 (+1174260us) service_pool.cc:227] Handling call
0808 01:11:00.049345 (+ 33070us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:00.137720 (+ 88375us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:00.137962 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:00.211615 1565 leader_election.cc:334] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.848s (SENT)
I20260808 01:11:00.212102 1565 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:00.163578 3929 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:00.138069 1487 leader_election.cc:334] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.000s (SENT)
I20260808 01:11:00.216779 1487 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:00.217941 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 79910.8us
I20260808 01:11:00.044010 3865 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:00.103695 4040 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:00.048377 4045 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:00.231371 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 394022us
I20260808 01:11:00.153182 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:00.236442 1594 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:00.237519 1489 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:00.156999 1444 raft_consensus.cc:1226] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:00.070976 3992 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:00.067884 3783 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:00.167156 2171 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:00.173990 2201 raft_consensus.cc:1226] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEADER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:00.240512 2201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 950) took 3455 ms. Trace:
I20260808 01:11:00.240856 2201 rpcz_store.cc:276] 0808 01:10:56.785533 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.148614 (+3363081us) service_pool.cc:227] Handling call
0808 01:11:00.173716 (+ 25102us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:00.240271 (+ 66555us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:00.240457 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:00.242575 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:00.243068 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:00.173804 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:00.137663 4014 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:00.048480 4065 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:00.141958 4120 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:00.148474 3911 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:00.188259 1447 raft_consensus.cc:2379] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 3: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:00.109318 4087 consensus_queue.cc:1046] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260808 01:11:00.144903 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:00.166615 4110 log.cc:924] Time spent T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.175s user 0.001s sys 0.000s
I20260808 01:11:00.230175 3806 raft_consensus.cc:2726] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:00.103564 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:00.200603 1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2465) took 1910 ms. Trace:
I20260808 01:11:00.274096 1602 rpcz_store.cc:276] 0808 01:10:58.290583 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.915096 (+624513us) service_pool.cc:227] Handling call
0808 01:10:58.915409 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:58.915624 (+ 215us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 01:10:59.832258 (+916634us) log.cc:842] Serialized 14 byte log entry
0808 01:11:00.200244 (+367986us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:00.200349 (+ 105us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:00.200397 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:00.200456 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:00.200551 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":974719,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":361,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":29585,"thread_start_us":1252198,"threads_started":2}
I20260808 01:11:00.206527 2200 raft_consensus.cc:1226] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:00.192014 3801 raft_consensus.cc:686] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:00.248800 3929 raft_consensus.cc:2781] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election won for term 6
W20260808 01:11:00.066431 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 219571us
I20260808 01:11:00.156009 4118 raft_consensus.cc:2726] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:00.137116 3861 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:00.229219 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 157590us
I20260808 01:11:00.148293 4082 consensus_queue.cc:1046] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.052s
I20260808 01:11:00.199280 1438 pending_rounds.cc:97] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20260808 01:11:00.187762 3840 raft_consensus.cc:686] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:00.243366 1524 raft_consensus.cc:2379] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 3: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:00.181109 2207 raft_consensus.cc:2396] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:11:00.247432 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:00.193773 4067 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:00.205997 4041 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:00.154052 4015 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:00.278398 4109 log.cc:924] Time spent T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.307s user 0.000s sys 0.002s
I20260808 01:11:00.261051 2204 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEARNER]: Advancing to term 5
I20260808 01:11:00.131450 4086 consensus_queue.cc:260] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:00.196559 1490 leader_election.cc:334] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.775s (SENT)
I20260808 01:11:00.313304 1490 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:00.216107 3963 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:00.276933 2200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 963) took 3272 ms. Trace:
I20260808 01:11:00.193678 3689 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:00.186977 3795 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:00.244690 3907 raft_consensus.cc:2781] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:11:00.279124 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:00.132061 4084 consensus_queue.cc:260] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:00.213696 1565 leader_election.cc:334] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.914s (SENT)
I20260808 01:11:00.344370 1565 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:00.283704 1487 leader_election.cc:334] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.116s (SENT)
I20260808 01:11:00.345292 1487 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:00.236205 4113 raft_consensus.cc:2726] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:00.213025 4057 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:00.307031 1522 raft_consensus.cc:2454] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:11:00.318437 2200 rpcz_store.cc:276] 0808 01:10:57.004389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.166445 (+3162056us) service_pool.cc:227] Handling call
0808 01:11:00.206263 (+ 39818us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:00.276590 (+ 70327us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:00.276861 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:00.321964 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 125447us
W20260808 01:11:00.360955 1490 leader_election.cc:334] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 2.187s (SENT)
I20260808 01:11:00.361490 1490 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:00.324509 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:00.329480 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:00.334091 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 119147us
I20260808 01:11:00.246608 4040 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:00.302692 4099 raft_consensus.cc:2726] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:00.250957 4014 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:00.306140 1438 raft_consensus.cc:2866] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:00.307873 4116 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:00.338873 3979 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:00.353986 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 70335.6us
W20260808 01:11:00.389500 1487 leader_election.cc:334] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.772s (SENT)
W20260808 01:11:00.390196 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 86974.3us
I20260808 01:11:00.340375 4114 raft_consensus.cc:2726] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:00.354946 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:00.345942 4031 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:00.400677 1487 leader_election.cc:334] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.243s (SENT)
I20260808 01:11:00.267014 4120 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:00.303150 3806 consensus_queue.cc:1046] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260808 01:11:00.368583 1444 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:11:00.350441 2203 raft_consensus.cc:2396] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:11:00.365492 3801 consensus_queue.cc:237] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:00.449241 4125 log.cc:924] Time spent T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.169s user 0.000s sys 0.001s
I20260808 01:11:00.334908 4042 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:00.381421 1446 raft_consensus.cc:2379] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 3: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:00.381438 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:00.395081 1448 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:00.453191 3689 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:00.453675 4122 log.cc:924] Time spent T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260808 01:11:00.436401 3816 raft_consensus.cc:2726] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:11:00.454614 3816 raft_consensus.cc:3032] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:11:00.365695 4124 consensus_queue.cc:1046] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260808 01:11:00.314823 3929 raft_consensus.cc:686] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:00.386121 3907 raft_consensus.cc:686] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:00.420462 2204 raft_consensus.cc:1261] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20260808 01:11:00.436046 2200 raft_consensus.cc:1261] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:11:00.443133 1518 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:11:00.388334 3748 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:00.453836 3772 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:00.400665 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 186984us
I20260808 01:11:00.315061 4041 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:00.392438 4084 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:00.491158 2204 pending_rounds.cc:76] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd: Aborting all ops after (but not including) 4
I20260808 01:11:00.494858 2204 pending_rounds.cc:97] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20260808 01:11:00.495234 2204 raft_consensus.cc:2866] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEARNER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:00.491815 2200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 969) took 3484 ms. Trace:
I20260808 01:11:00.500444 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 966) took 3494 ms. Trace:
I20260808 01:11:00.501451 2204 rpcz_store.cc:276] 0808 01:10:57.005567 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.237841 (+3232274us) service_pool.cc:227] Handling call
0808 01:11:00.260903 (+ 23062us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:00.500113 (+239210us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:00.500368 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:00.496111 2200 rpcz_store.cc:276] 0808 01:10:57.007338 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.360714 (+3353376us) service_pool.cc:227] Handling call
0808 01:11:00.406972 (+ 46258us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:00.491438 (+ 84466us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:00.491732 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:00.490358 1448 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
W20260808 01:11:00.500617 1565 leader_election.cc:334] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.027s (SENT)
I20260808 01:11:00.404882 4113 raft_consensus.cc:2726] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:00.416033 4009 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:00.455127 3816 raft_consensus.cc:729] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:00.532531 1600 raft_consensus.cc:2379] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 6: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:00.468281 4117 raft_consensus.cc:2726] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:00.442313 4115 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:00.469748 3865 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:00.471167 3795 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:00.469477 4057 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:00.339147 4121 consensus_queue.cc:1046] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.153s
I20260808 01:11:00.468701 3818 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:00.467638 4120 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:00.563161 1523 raft_consensus.cc:1226] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
W20260808 01:11:00.565270 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 168721us
I20260808 01:11:00.568579 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:00.441130 4031 consensus_queue.cc:1046] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.113s
I20260808 01:11:00.570643 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:00.573105 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:00.554139 4045 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:00.553824 3944 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:00.536909 4131 raft_consensus.cc:2726] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:11:00.415891 3840 consensus_queue.cc:237] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:00.596073 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 232629us
W20260808 01:11:00.609827 1490 leader_election.cc:334] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.855s (SENT)
I20260808 01:11:00.610476 1490 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:00.570299 3783 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:00.573662 3992 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:11:00.615391 1490 leader_election.cc:334] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 2.158s (SENT)
I20260808 01:11:00.616456 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:00.401240 4116 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:00.620486 1490 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:00.584800 4120 raft_consensus.cc:3032] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:11:00.624713 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:00.523258 3772 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:00.627099 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:00.570464 4124 consensus_queue.cc:1046] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260808 01:11:00.633896 1598 raft_consensus.cc:1261] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:11:00.623306 3943 raft_consensus.cc:2726] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:00.638114 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:00.645612 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:00.585274 4009 consensus_queue.cc:1046] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
W20260808 01:11:00.647927 1597 consensus_meta.cc:287] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.511s user 0.012s sys 0.002s
I20260808 01:11:00.654031 1447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55042 (request call id 2309) took 1972 ms. Trace:
W20260808 01:11:00.667542 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 500287us
I20260808 01:11:00.668546 1447 rpcz_store.cc:276] 0808 01:10:58.681344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.819617 (+1138273us) service_pool.cc:227] Handling call
0808 01:11:00.653801 (+834184us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b540002e9d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:00.653956 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":750065920}
I20260808 01:11:00.668968 2171 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:00.618682 3907 consensus_queue.cc:237] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:00.626379 4045 raft_consensus.cc:2726] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:11:00.656965 1443 raft_consensus.cc:1226] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:00.657358 3748 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:00.658254 2207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 550) took 1466 ms. Trace:
I20260808 01:11:00.610496 3929 consensus_queue.cc:237] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:00.623196 4113 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:00.626019 3861 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:00.627288 4120 raft_consensus.cc:729] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:00.662340 4031 raft_consensus.cc:3032] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:11:00.662369 1524 raft_consensus.cc:2379] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:00.665148 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:00.668085 1597 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:11:00.572638 4133 consensus_queue.cc:1035] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.091s
I20260808 01:11:00.612424 3931 raft_consensus.cc:686] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:00.670734 4082 raft_consensus.cc:3032] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:00.679859 4009 consensus_queue.cc:1225] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer 6b09dbefd24b4d52b385c4dff5697dc5 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260808 01:11:00.674082 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:00.673522 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:00.709743 1598 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:00.564224 4115 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:00.677876 2207 rpcz_store.cc:276] 0808 01:10:59.192203 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.069807 (+877604us) service_pool.cc:227] Handling call
0808 01:11:00.657983 (+588176us) spinlock_profiling.cc:243] Waited 40.3 ms on lock 0x7b540067d254. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:00.658173 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":92715392}
I20260808 01:11:00.679636 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:00.772753 1602 raft_consensus.cc:2379] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 3: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:00.688943 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 719) took 3842 ms. Trace:
I20260808 01:11:00.683156 2171 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:00.696476 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:00.707382 1604 raft_consensus.cc:1261] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260808 01:11:00.707505 1565 leader_election.cc:334] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.167s (SENT)
I20260808 01:11:00.673503 4015 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:00.693660 4014 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:00.773869 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:00.887194 1524 raft_consensus.cc:2379] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 4: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:00.853770 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:00.887921 1594 raft_consensus.cc:2454] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:11:00.888029 2206 rpcz_store.cc:276] 0808 01:10:56.846422 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.166000 (+3319578us) service_pool.cc:227] Handling call
0808 01:11:00.198815 (+ 32815us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:00.199082 (+ 267us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 01:11:00.602768 (+403686us) log.cc:842] Serialized 14 byte log entry
0808 01:11:00.688527 (+ 85759us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:00.688582 (+ 55us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:00.688671 (+ 89us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:00.688739 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:00.688878 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2031,"prepare.queue_time_us":493745,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":2170,"raft.run_wall_time_us":4503,"spinlock_wait_cycles":5100416,"thread_start_us":431662,"threads_started":2}
I20260808 01:11:00.771178 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:11:00.889111 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 181641us
I20260808 01:11:00.890202 1603 raft_consensus.cc:2379] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 4: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:00.691920 4124 raft_consensus.cc:2781] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:00.803737 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:00.847193 2207 raft_consensus.cc:1226] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEARNER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:00.898046 2207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 722) took 1602 ms. Trace:
I20260808 01:11:00.898612 2207 rpcz_store.cc:276] 0808 01:10:59.295170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.772226 (+1477056us) service_pool.cc:227] Handling call
0808 01:11:00.846934 (+ 74708us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:00.897756 (+ 50822us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:00.897982 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:00.900242 2207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 724) took 1600 ms. Trace:
I20260808 01:11:00.904094 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:00.894281 1598 raft_consensus.cc:2454] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:00.931571 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:00.933475 1602 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
W20260808 01:11:00.934835 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 251769us
I20260808 01:11:00.937304 2171 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:00.900758 2207 rpcz_store.cc:276] 0808 01:10:59.300232 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.899655 (+1599423us) service_pool.cc:227] Handling call
0808 01:11:00.900177 (+ 522us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:00.974761 1447 raft_consensus.cc:1226] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:00.976151 4115 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:00.978299 3911 consensus_meta.cc:287] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.710s user 0.005s sys 0.009s
I20260808 01:11:00.979535 1517 raft_consensus.cc:1226] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:00.982439 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:01.000263 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 63092.3us
I20260808 01:11:00.709316 4082 raft_consensus.cc:729] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:01.006306 1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2475) took 1169 ms. Trace:
I20260808 01:11:01.000203 1602 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:01.009371 1596 rpcz_store.cc:276] 0808 01:10:59.836507 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.836632 (+ 125us) service_pool.cc:227] Handling call
0808 01:10:59.837506 (+ 874us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:10:59.980377 (+142871us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.006030 (+1025653us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b540018ddd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:01.006240 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":320092544}
I20260808 01:11:01.008826 1517 raft_consensus.cc:1226] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
W20260808 01:11:01.015467 1410 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.000s sys 0.028s
W20260808 01:11:01.015964 1410 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.624s user 0.000s sys 0.029s
W20260808 01:11:01.016391 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 625063us
I20260808 01:11:01.018154 1410 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:00.894758 4124 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:01.018498 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:01.040134 1517 raft_consensus.cc:2379] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:00.693750 4031 raft_consensus.cc:729] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:01.022948 1524 raft_consensus.cc:1261] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260808 01:11:01.026062 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 46493us
W20260808 01:11:01.046332 1408 consensus_peers.cc:487] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d -> Peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Couldn't send request to peer 95b661b03f1d460b9657e61d4b0247cd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:11:01.046653 1410 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:01.036087 1443 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 1999) took 2329 ms. Trace:
I20260808 01:11:01.047688 1443 rpcz_store.cc:276] 0808 01:10:58.706308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.988669 (+1282361us) service_pool.cc:227] Handling call
0808 01:11:00.656742 (+668073us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:00.680639 (+ 23897us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.035701 (+355062us) spinlock_profiling.cc:243] Waited 668 ms on lock 0x7b540002e9d0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:01.035989 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1535550336}
W20260808 01:11:00.998832 4143 log.cc:924] Time spent T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.086s user 0.001s sys 0.001s
I20260808 01:11:01.048750 1410 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:00.932660 3795 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:01.042150 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 723) took 1746 ms. Trace:
I20260808 01:11:01.055815 2206 rpcz_store.cc:276] 0808 01:10:59.295494 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.889969 (+1594475us) service_pool.cc:227] Handling call
0808 01:11:00.940171 (+ 50202us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:01.036732 (+ 96561us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:01.036855 (+ 123us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:01.041388 (+ 4533us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:01.041546 (+ 158us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.041738 (+ 192us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:01.042035 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61652,"raft.queue_time_us":111842,"thread_start_us":33852,"threads_started":1}
I20260808 01:11:01.078086 1596 raft_consensus.cc:1226] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:01.040454 3748 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:01.037922 4030 raft_consensus.cc:2726] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:01.043776 4124 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:01.058555 4087 raft_consensus.cc:2781] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:01.051613 4139 raft_consensus.cc:2781] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:01.113422 1602 raft_consensus.cc:1226] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: []
W20260808 01:11:01.096525 3990 consensus_meta.cc:287] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.898s user 0.014s sys 0.000s
W20260808 01:11:01.096246 3970 consensus_meta.cc:287] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.889s user 0.014s sys 0.002s
I20260808 01:11:00.699383 4113 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:01.141013 1442 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2000) took 2226 ms. Trace:
I20260808 01:11:01.042965 3911 consensus_queue.cc:1046] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.070s
I20260808 01:11:00.934260 4120 consensus_queue.cc:260] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:01.071813 3795 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:01.147008 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 725) took 1824 ms. Trace:
I20260808 01:11:01.239930 1598 raft_consensus.cc:1226] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:01.201328 4014 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:11:01.236393 4143 log.cc:924] Time spent T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260808 01:11:01.240937 3911 consensus_queue.cc:1225] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer 6b09dbefd24b4d52b385c4dff5697dc5 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260808 01:11:01.241508 3970 raft_consensus.cc:2726] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:01.107642 3931 consensus_queue.cc:237] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:01.288645 4153 log.cc:924] Time spent T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.193s user 0.001s sys 0.000s
W20260808 01:11:01.290414 4155 log.cc:924] Time spent T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.149s user 0.000s sys 0.000s
I20260808 01:11:01.114065 3748 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:01.147915 4087 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:01.292317 4087 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:01.292889 3748 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:01.264714 1448 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55036
I20260808 01:11:01.295089 1448 raft_consensus.cc:595] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Received request to transfer leadership
W20260808 01:11:01.243454 1525 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58266 (request call id 2071) took 1405 ms (client timeout 1870 ms). Trace:
I20260808 01:11:01.147948 2207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 562) took 1825 ms. Trace:
I20260808 01:11:01.242259 1442 rpcz_store.cc:276] 0808 01:10:58.914241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.053999 (+1139758us) service_pool.cc:227] Handling call
0808 01:11:00.097006 (+ 43007us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:00.130366 (+ 33360us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:00.130401 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:00.130442 (+ 41us) raft_consensus.cc:1631] Marking committed up to 2
0808 01:11:00.130498 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.140649 (+1010151us) spinlock_profiling.cc:243] Waited 33.1 ms on lock 0x7b540014d2d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:01.140719 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:01.140929 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":76208000}
I20260808 01:11:01.096822 4145 consensus_queue.cc:1046] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.192s
I20260808 01:11:01.239987 2206 rpcz_store.cc:276] 0808 01:10:59.322844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:01.062524 (+1739680us) service_pool.cc:227] Handling call
0808 01:11:01.145661 (+ 83137us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:01.146114 (+ 453us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:01.146418 (+ 304us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:01.146525 (+ 107us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:01.146592 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.146684 (+ 92us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:01.146929 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:01.080147 4146 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:11:01.279340 1596 raft_consensus.cc:1226] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:01.147346 4139 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:01.282851 3772 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:01.284687 4065 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:01.286108 4130 log.cc:924] Time spent T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23: Append to log took a long time: real 0.816s user 0.004s sys 0.006s
I20260808 01:11:01.309448 3911 raft_consensus.cc:3032] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:11:01.296823 2207 rpcz_store.cc:276] 0808 01:10:59.322544 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.954034 (+1631490us) service_pool.cc:227] Handling call
0808 01:11:00.954412 (+ 378us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:01.147508 (+193096us) raft_consensus.cc:1520] Early marking committed up to index 5
0808 01:11:01.147540 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:01.147583 (+ 43us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:01.147624 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.147699 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:01.147859 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54558,"raft.queue_time_us":155929,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":30547,"thread_start_us":138792,"threads_started":1}
W20260808 01:11:01.298030 1525 rpcz_store.cc:269] 0808 01:10:59.838102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.838242 (+ 140us) service_pool.cc:227] Handling call
0808 01:11:01.243194 (+1404952us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b540037ebd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:01.243377 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":246153984}
I20260808 01:11:01.150961 4156 raft_consensus.cc:2781] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election won for term 3
W20260808 01:11:01.292335 4154 log.cc:924] Time spent T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.092s user 0.000s sys 0.001s
I20260808 01:11:01.309901 3911 raft_consensus.cc:729] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:01.322223 1516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2073) took 1459 ms. Trace:
I20260808 01:11:01.322808 1516 rpcz_store.cc:276] 0808 01:10:59.863649 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.863895 (+ 246us) service_pool.cc:227] Handling call
0808 01:10:59.929756 (+ 65861us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:00.129963 (+200207us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:00.130143 (+ 180us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:00.131117 (+ 974us) log.cc:842] Serialized 26 byte log entry
0808 01:11:00.571910 (+440793us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:00.605304 (+ 33394us) log.cc:842] Serialized 14 byte log entry
0808 01:11:00.606012 (+ 708us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:00.606120 (+ 108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:01.319349 (+713229us) raft_consensus.cc:1671] finished
0808 01:11:01.321958 (+ 2609us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:01.322141 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":32321,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":86758,"raft.run_cpu_time_us":376,"raft.run_wall_time_us":9305,"thread_start_us":574066,"threads_started":2,"wal-append.queue_time_us":569132}
I20260808 01:11:01.329697 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:01.273123 4030 consensus_queue.cc:1046] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260808 01:11:01.336478 1567 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:01.340404 1596 raft_consensus.cc:1226] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:01.312220 4139 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:11:01.347019 1565 leader_election.cc:334] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 3.224s (SENT)
I20260808 01:11:01.345640 1523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2070) took 1509 ms. Trace:
I20260808 01:11:01.345844 1567 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:01.355242 1598 raft_consensus.cc:1226] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:01.355186 1567 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:01.342170 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 726) took 2013 ms. Trace:
I20260808 01:11:01.353252 1523 rpcz_store.cc:276] 0808 01:10:59.836538 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.837772 (+ 1234us) service_pool.cc:227] Handling call
0808 01:10:59.889077 (+ 51305us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:00.564539 (+675462us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.345322 (+780783us) spinlock_profiling.cc:243] Waited 674 ms on lock 0x7b54000df054. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:01.345556 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1549571840}
I20260808 01:11:01.356424 2206 rpcz_store.cc:276] 0808 01:10:59.329183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:01.298205 (+1969022us) service_pool.cc:227] Handling call
0808 01:11:01.341507 (+ 43302us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:01.341741 (+ 234us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:01.341779 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:01.341845 (+ 66us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:01.341891 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.341959 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:01.342099 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:01.296469 4014 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:01.317466 4146 raft_consensus.cc:686] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:01.174553 4031 consensus_queue.cc:260] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:01.148452 4082 consensus_queue.cc:260] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:01.360850 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:01.387697 1525 raft_consensus.cc:2379] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:01.344916 3748 raft_consensus.cc:2726] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:11:01.385633 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:01.398926 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:01.388687 1596 raft_consensus.cc:1226] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260808 01:11:01.331322 3911 consensus_queue.cc:260] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:01.054015 2205 raft_consensus.cc:2932] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:01.354287 4140 raft_consensus.cc:2726] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:01.389001 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 729) took 1918 ms. Trace:
I20260808 01:11:01.408089 2206 rpcz_store.cc:276] 0808 01:10:59.470731 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:01.357753 (+1887022us) service_pool.cc:227] Handling call
0808 01:11:01.388463 (+ 30710us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:01.388635 (+ 172us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:01.388663 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:01.388703 (+ 40us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:01.388743 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.388797 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:01.388939 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:01.411689 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:01.412951 1523 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:01.414801 3911 raft_consensus.cc:3037] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:01.365691 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:01.390854 4067 consensus_queue.cc:1035] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:11:01.394481 4113 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:01.418481 1516 raft_consensus.cc:2379] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 3: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:01.396742 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:01.420647 4152 log.cc:924] Time spent T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260808 01:11:01.422236 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 720) took 4179 ms. Trace:
I20260808 01:11:01.375949 3783 consensus_queue.cc:1046] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260808 01:11:01.422744 2204 rpcz_store.cc:276] 0808 01:10:57.242960 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.502597 (+3259637us) service_pool.cc:227] Handling call
0808 01:11:00.503252 (+ 655us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:00.524448 (+ 21196us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:00.524491 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:00.565029 (+ 40538us) log.cc:842] Serialized 26 byte log entry
0808 01:11:00.846015 (+280986us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:00.853001 (+ 6986us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:00.853109 (+ 108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:01.421948 (+568839us) raft_consensus.cc:1671] finished
0808 01:11:01.421981 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:01.422152 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20692,"raft.queue_time_us":120961,"raft.run_cpu_time_us":809,"raft.run_wall_time_us":19207,"thread_start_us":145944,"threads_started":1,"wal-append.queue_time_us":693103}
I20260808 01:11:01.426143 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 731) took 1627 ms. Trace:
I20260808 01:11:01.426667 2204 rpcz_store.cc:276] 0808 01:10:59.799178 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:01.425026 (+1625848us) service_pool.cc:227] Handling call
0808 01:11:01.425464 (+ 438us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:01.425711 (+ 247us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:01.425750 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:01.425808 (+ 58us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:01.425858 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.425941 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:01.426074 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:01.429066 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:01.413070 3802 consensus_meta.cc:287] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 1.427s user 0.019s sys 0.002s
I20260808 01:11:01.430377 2202 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:01.430907 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:01.431213 4126 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:01.431748 1439 raft_consensus.cc:2379] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:01.431805 1447 raft_consensus.cc:1226] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260808 01:11:01.432331 1525 raft_consensus.cc:1261] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:11:01.431748 4126 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:01.433382 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:01.340472 4156 raft_consensus.cc:686] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:01.436275 4156 consensus_queue.cc:237] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:01.437093 4099 consensus_queue.cc:1035] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:11:01.440632 4146 consensus_queue.cc:237] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:01.430456 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:01.420671 1516 raft_consensus.cc:1261] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:11:01.453504 1489 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:01.455902 1596 raft_consensus.cc:1226] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260808 01:11:01.470742 1568 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:01.494287 1442 raft_consensus.cc:1261] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:11:01.482847 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:01.503803 1447 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:11:01.377406 4119 consensus_queue.cc:1046] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.104s
I20260808 01:11:01.368983 4144 raft_consensus.cc:2726] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:01.482875 1523 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:11:01.480762 4082 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:01.479666 4045 raft_consensus.cc:2781] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:11:01.501518 3748 raft_consensus.cc:2781] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:01.502135 4031 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:01.480470 4099 consensus_queue.cc:1035] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260808 01:11:01.494062 4102 consensus_queue.cc:1046] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
W20260808 01:11:01.531212 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 452467us
I20260808 01:11:01.522675 4045 raft_consensus.cc:686] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:01.522300 3795 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:11:01.538522 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 495137us
W20260808 01:11:01.540352 1519 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.003s sys 0.008s
W20260808 01:11:01.541349 1519 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.530s user 0.004s sys 0.008s
W20260808 01:11:01.541745 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 486680us
I20260808 01:11:01.527737 3748 raft_consensus.cc:686] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:01.523350 4119 raft_consensus.cc:3032] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:01.543412 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:01.519807 4144 consensus_queue.cc:1046] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260808 01:11:01.557750 2171 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:01.559453 2171 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
W20260808 01:11:01.540351 4084 consensus_meta.cc:287] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 1.042s user 0.015s sys 0.000s
W20260808 01:11:01.569556 1410 connection.cc:232] RPC call timeout handler was delayed by -2.351323246024549s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2331s for a response
I20260808 01:11:01.571106 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:01.563536 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 730) took 1999 ms. Trace:
I20260808 01:11:01.566417 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:01.567745 1602 raft_consensus.cc:1261] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:11:01.576668 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:01.576897 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:01.577658 1594 raft_consensus.cc:2362] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4. Re-sending same reply.
I20260808 01:11:01.578729 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:01.579149 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:01.580160 1594 raft_consensus.cc:2379] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 6: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:01.580883 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:01.564724 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 225801us
W20260808 01:11:01.582175 1490 leader_election.cc:334] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.943s (SENT)
I20260808 01:11:01.577469 1604 raft_consensus.cc:2362] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6. Re-sending same reply.
I20260808 01:11:01.583787 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:01.584668 1599 raft_consensus.cc:2396] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260808 01:11:01.586092 1602 raft_consensus.cc:1226] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20260808 01:11:01.579730 1596 raft_consensus.cc:2362] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5. Re-sending same reply.
I20260808 01:11:01.582687 1490 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:01.589383 1490 leader_election.cc:334] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 2.035s (SENT)
W20260808 01:11:01.590401 1490 leader_election.cc:334] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 2.088s (SENT)
W20260808 01:11:01.591655 1490 leader_election.cc:334] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 2.241s (SENT)
I20260808 01:11:01.592125 1490 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:01.593355 1490 leader_election.cc:334] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 2.334s (SENT)
W20260808 01:11:01.594911 1490 leader_election.cc:334] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 2.772s (SENT)
I20260808 01:11:01.595359 1490 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:01.566994 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:01.596575 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:01.572983 2206 rpcz_store.cc:276] 0808 01:10:59.564607 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:01.409531 (+1844924us) service_pool.cc:227] Handling call
0808 01:11:01.410031 (+ 500us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:01.410325 (+ 294us) raft_consensus.cc:1520] Early marking committed up to index 8
0808 01:11:01.411093 (+ 768us) log.cc:842] Serialized 14 byte log entry
0808 01:11:01.563236 (+152143us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:01.563280 (+ 44us) raft_consensus.cc:1631] Marking committed up to 8
0808 01:11:01.563324 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.563379 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:01.563487 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3763,"prepare.queue_time_us":213,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":50,"raft.queue_time_us":5843,"thread_start_us":147417,"threads_started":1}
W20260808 01:11:01.598709 1410 leader_election.cc:334] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 1.919s (SENT)
W20260808 01:11:01.603051 1410 leader_election.cc:334] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 1.980s (SENT)
I20260808 01:11:01.599063 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:01.558185 4119 raft_consensus.cc:729] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:01.607419 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:01.607779 1525 raft_consensus.cc:1261] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260808 01:11:01.604141 1410 leader_election.cc:334] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 2.028s (SENT)
I20260808 01:11:01.609966 1410 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:01.611497 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:01.610069 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:01.562935 4170 raft_consensus.cc:2726] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:01.583148 4065 raft_consensus.cc:2781] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:11:01.621479 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:01.542732 4161 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.065s
I20260808 01:11:01.617311 1446 raft_consensus.cc:1226] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
W20260808 01:11:01.617149 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 144047us
I20260808 01:11:01.610996 3929 raft_consensus.cc:2726] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:11:01.629602 1568 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:01.618566 1567 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:01.621299 1520 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58280 (request call id 1852) took 3414 ms (client timeout 2611 ms). Trace:
W20260808 01:11:01.630645 1520 rpcz_store.cc:269] 0808 01:10:58.202572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.275761 (+ 73189us) service_pool.cc:227] Handling call
0808 01:10:58.760143 (+484382us) spinlock_profiling.cc:243] Waited 5.89 ms on lock 0x7b5800035d98. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adf851 00007fe349adfbce 000000000034aea9 00007fe35190ba30 00007fe3519d17b4 00007fe3519d1821 00007fe3519811b0 00007fe351980a94 00007fe34b068c04 00007fe34b068b92
0808 01:11:01.616734 (+2856591us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540035d4d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:01.616915 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2428657920}
I20260808 01:11:01.622330 4065 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:01.632166 1520 raft_consensus.cc:1226] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:01.632231 4065 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:01.610538 4014 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:01.635344 1568 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:01.636121 1447 raft_consensus.cc:1226] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 7. Current term is 8. Ops: []
W20260808 01:11:01.621522 1490 leader_election.cc:334] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 2.628s (SENT)
I20260808 01:11:01.600060 3772 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:01.621824 1487 leader_election.cc:334] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.724s (SENT)
I20260808 01:11:01.616250 1603 raft_consensus.cc:2362] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3. Re-sending same reply.
I20260808 01:11:01.597059 4102 raft_consensus.cc:2726] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:01.630074 2206 raft_consensus.cc:2396] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260808 01:11:01.655371 1487 leader_election.cc:334] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.744s (SENT)
W20260808 01:11:01.654088 1565 leader_election.cc:334] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.731s (SENT)
W20260808 01:11:01.656711 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 52576.1us
I20260808 01:11:01.658999 1565 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:01.661177 1410 leader_election.cc:334] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 2.151s (SENT)
I20260808 01:11:01.668818 1410 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:01.670648 1490 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:01.671362 4187 raft_consensus.cc:2726] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:01.673190 4179 raft_consensus.cc:2726] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:11:01.674057 4179 raft_consensus.cc:2726] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:01.675027 1438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55042 (request call id 2268) took 5604 ms. Trace:
I20260808 01:11:01.624284 3748 consensus_queue.cc:237] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:01.674767 4179 consensus_queue.cc:1046] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:01.681165 1490 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:01.653735 4014 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:01.621683 4045 consensus_queue.cc:237] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:01.661010 3840 raft_consensus.cc:2726] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:01.692178 4014 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:01.688715 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:01.696982 1520 raft_consensus.cc:2454] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:11:01.653311 4119 consensus_queue.cc:260] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:01.698747 4119 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:01.654054 3783 raft_consensus.cc:2726] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:01.655629 3931 raft_consensus.cc:2726] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:01.708563 1490 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:01.710930 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 93164.5us
I20260808 01:11:01.695569 4014 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:01.717516 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:01.718395 1520 raft_consensus.cc:2454] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:11:01.719502 1442 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55036
I20260808 01:11:01.720114 1442 raft_consensus.cc:595] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Received request to transfer leadership
I20260808 01:11:01.721093 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:01.722208 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:01.722807 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:01.723410 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:01.703191 3929 raft_consensus.cc:2781] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:01.729104 1443 raft_consensus.cc:1226] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260808 01:11:01.730072 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:01.731920 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:01.732219 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:01.733103 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:01.716106 2171 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:01.730903 1439 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:11:01.736256 1567 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:01.737815 2171 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:01.701579 4184 raft_consensus.cc:2726] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:01.736645 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:01.740474 1448 raft_consensus.cc:3032] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:01.741009 1448 raft_consensus.cc:729] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:01.744043 1444 raft_consensus.cc:3032] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:11:01.744596 1444 raft_consensus.cc:729] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:01.660930 3772 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:01.756036 2205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 968) took 4749 ms. Trace:
I20260808 01:11:01.675483 1438 rpcz_store.cc:276] 0808 01:10:56.070263 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.082629 (+2012366us) service_pool.cc:227] Handling call
0808 01:10:58.134778 (+ 52149us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:00.439000 (+2304222us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.666708 (+1227708us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b540014d7d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:01.674951 (+ 8243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4224731520}
I20260808 01:11:01.736040 4178 consensus_queue.cc:1035] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.016s
I20260808 01:11:01.714954 4009 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:01.753078 4036 consensus_queue.cc:1035] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:11:01.754228 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:01.754455 4169 log.cc:924] Time spent T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.069s user 0.000s sys 0.001s
I20260808 01:11:01.712922 4132 raft_consensus.cc:2726] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:11:01.757192 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:01.747686 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:01.777076 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 135147us
I20260808 01:11:01.777479 1525 raft_consensus.cc:1226] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:01.758118 2171 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:01.750505 1444 consensus_queue.cc:260] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:01.761516 4150 raft_consensus.cc:2726] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:01.750507 4185 consensus_queue.cc:1035] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
W20260808 01:11:01.763258 4174 log.cc:924] Time spent T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20260808 01:11:01.766991 4136 raft_consensus.cc:3032] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:11:01.791052 1516 raft_consensus.cc:1226] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:01.769289 3799 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:01.769964 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 141394us
I20260808 01:11:01.742113 1448 consensus_queue.cc:260] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:01.777056 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:01.791294 4136 raft_consensus.cc:729] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:01.761309 3772 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:01.759724 4014 raft_consensus.cc:2781] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:01.788775 1444 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:11:01.773551 4124 consensus_meta.cc:287] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.621s user 0.012s sys 0.000s
I20260808 01:11:01.792168 2171 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:01.800748 1448 raft_consensus.cc:3037] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:01.773852 4120 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:01.775604 4102 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:01.805569 1410 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:01.789902 3806 raft_consensus.cc:2781] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election won for term 3
W20260808 01:11:01.817097 1601 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2166) took 3625 ms (client timeout 2016 ms). Trace:
I20260808 01:11:01.791266 4161 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:01.823468 3816 consensus_queue.cc:260] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:01.826984 4086 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:01.826833 4186 consensus_queue.cc:1035] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260808 01:11:01.812143 3911 raft_consensus.cc:2781] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:01.797049 4170 raft_consensus.cc:2781] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 01:11:01.830430 1601 rpcz_store.cc:269] 0808 01:10:58.191217 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:58.314731 (+123514us) service_pool.cc:227] Handling call
0808 01:11:01.816845 (+3502114us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b54002dd7d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:01.817009 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4070173056}
W20260808 01:11:01.840233 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 131736us
I20260808 01:11:01.816217 3783 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:01.846943 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:01.857767 4173 log.cc:924] Time spent T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.148s user 0.000s sys 0.000s
W20260808 01:11:01.858098 4182 log.cc:924] Time spent T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20260808 01:11:01.843264 3806 raft_consensus.cc:686] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
W20260808 01:11:01.860767 4191 log.cc:924] Time spent T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.057s user 0.002s sys 0.000s
I20260808 01:11:01.860572 4040 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:01.862906 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:01.899667 3722 log.cc:924] Time spent T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20260808 01:11:01.871297 1444 raft_consensus.cc:1261] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:11:01.878970 1448 raft_consensus.cc:1261] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20260808 01:11:01.946616 1448 pending_rounds.cc:76] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d: Aborting all ops after (but not including) 3
I20260808 01:11:01.947001 1448 pending_rounds.cc:97] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20260808 01:11:01.947322 1448 raft_consensus.cc:2866] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:01.843945 4170 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:01.883266 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:01.885686 3915 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:11:01.756841 2205 rpcz_store.cc:276] 0808 01:10:57.007001 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.339355 (+3332354us) service_pool.cc:227] Handling call
0808 01:11:00.519633 (+180278us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:00.583856 (+ 64223us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:00.596992 (+ 13136us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:00.659046 (+ 62054us) log.cc:842] Serialized 564 byte log entry
0808 01:11:01.038233 (+379187us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:01.717935 (+679702us) log.cc:842] Serialized 14 byte log entry
0808 01:11:01.755300 (+ 37365us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.755483 (+ 183us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:01.755574 (+ 91us) raft_consensus.cc:1671] finished
0808 01:11:01.755616 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:01.755927 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13374,"raft.queue_time_us":20699,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":173,"thread_start_us":344634,"threads_started":1,"wal-append.queue_time_us":392606}
I20260808 01:11:01.882596 1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2482) took 1143 ms. Trace:
W20260808 01:11:01.897624 1487 leader_election.cc:334] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.615s (SENT)
W20260808 01:11:01.954511 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 56939.4us
W20260808 01:11:01.955870 1487 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.653s (SENT) [suppressed 76 similar messages]
W20260808 01:11:01.898144 2171 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:49260 dropped due to backpressure. The service queue is full; it has 50 items.
W20260808 01:11:01.921551 4176 log.cc:924] Time spent T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.339s user 0.002s sys 0.002s
I20260808 01:11:01.930962 4065 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:01.924377 4186 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:01.843389 3911 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:11:01.957571 3979 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260808 01:11:01.956044 4170 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:01.898770 3783 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:01.809437 4014 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:11:01.957010 1487 leader_election.cc:334] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.671s (SENT)
I20260808 01:11:01.953872 1597 rpcz_store.cc:276] 0808 01:11:00.739493 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.739704 (+ 211us) service_pool.cc:227] Handling call
0808 01:11:00.740259 (+ 555us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:00.928903 (+188644us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:00.928933 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:00.979328 (+ 50395us) log.cc:842] Serialized 26 byte log entry
0808 01:11:01.538389 (+559061us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:01.538430 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.538499 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:01.882353 (+343854us) raft_consensus.cc:1671] finished
0808 01:11:01.882374 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:01.882531 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":191654,"raft.queue_time_us":189058,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":65269,"thread_start_us":458196,"threads_started":1,"wal-append.queue_time_us":564716}
I20260808 01:11:01.809535 4136 consensus_queue.cc:260] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:01.964334 1520 raft_consensus.cc:1226] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:01.788237 4009 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:01.948875 4033 consensus_queue.cc:1046] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
W20260808 01:11:01.934880 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 54869.9us
I20260808 01:11:01.844542 4150 consensus_queue.cc:1046] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.122s
W20260808 01:11:01.973925 1565 leader_election.cc:334] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.718s (SENT)
I20260808 01:11:01.984269 1565 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:01.952472 3929 raft_consensus.cc:686] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:01.837718 4161 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:01.969619 3911 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:01.952594 1438 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:01.819095 4124 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:01.977403 4014 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:11:01.993592 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 322101us
W20260808 01:11:01.997205 1565 leader_election.cc:334] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.076s (SENT)
W20260808 01:11:01.983886 4087 consensus_meta.cc:287] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.691s user 0.014s sys 0.000s
I20260808 01:11:01.981578 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:01.995747 4040 raft_consensus.cc:2726] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:02.002893 4087 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:02.004192 2205 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:01.962433 2171 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:49260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1014)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1000)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1015)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1001)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1017)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1011)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1012)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1006)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1016)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 742)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1002)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1003)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 582)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 577)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 584)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1018)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1007)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 588)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1021)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1005)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1013)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 583)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 585)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 576)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 580)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1019)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1020)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1009)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 589)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1022)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 748)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1024)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 586)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 587)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 746)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 735)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 737)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 738)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 740)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 741)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 743)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1004)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1010)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 744)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1023)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 745)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 747)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 578)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 579)
I20260808 01:11:02.007915 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:02.008898 2205 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:02.010427 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:02.010869 1490 leader_election.cc:334] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.631s (SENT)
I20260808 01:11:02.012558 3795 raft_consensus.cc:3032] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:11:02.013034 3795 raft_consensus.cc:729] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:02.013926 3795 consensus_queue.cc:260] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:02.003530 3990 raft_consensus.cc:3032] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEADER]: Stepping down as leader of term 5
W20260808 01:11:02.020651 1490 leader_election.cc:334] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.724s (SENT)
I20260808 01:11:02.025091 1490 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:01.998257 4136 raft_consensus.cc:3037] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:02.014988 4009 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:02.043663 4116 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:02.019065 2171 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:49260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1001)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1017)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1011)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1012)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1006)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1016)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 742)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1002)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 591)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1003)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 582)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 577)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 584)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1018)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1007)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 588)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1021)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1005)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1013)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 583)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 585)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 576)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 580)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1019)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1020)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 590)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1009)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 589)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1022)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 748)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1024)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 586)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 587)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 594)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 746)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 735)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 737)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 738)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 740)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 741)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 743)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1004)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1010)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 744)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1023)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 745)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 747)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 578)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 579)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 592)
I20260808 01:11:02.046485 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:02.047356 2206 raft_consensus.cc:2396] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:11:02.027081 1568 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:02.029000 4188 raft_consensus.cc:2726] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:02.048453 4140 raft_consensus.cc:2781] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:02.028827 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:02.048888 4140 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:02.049501 4140 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:02.050493 3783 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:02.052896 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 732) took 2224 ms. Trace:
W20260808 01:11:02.041823 4139 consensus_meta.cc:287] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.676s user 0.011s sys 0.000s
I20260808 01:11:02.043300 4087 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:02.044348 3689 raft_consensus.cc:3032] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:11:02.029367 4124 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:02.046161 2171 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:49260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1017)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1011)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1012)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1006)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1016)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 742)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1002)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 591)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1003)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 582)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 577)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 584)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1018)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1007)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 588)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1021)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1005)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1013)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 583)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 585)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 576)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 580)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1019)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1020)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 590)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 595)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1009)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 589)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1022)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 748)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1024)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 586)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 587)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 594)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 746)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 735)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 737)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 738)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 740)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 741)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 743)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1004)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1010)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 744)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1023)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 745)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 747)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 578)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 579)
I20260808 01:11:02.053261 2204 rpcz_store.cc:276] 0808 01:10:59.828542 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:01.427979 (+1599437us) service_pool.cc:227] Handling call
0808 01:11:01.428453 (+ 474us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:01.429293 (+ 840us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:01.429324 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:01.444122 (+ 14798us) log.cc:842] Serialized 26 byte log entry
0808 01:11:01.558532 (+114410us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:01.558571 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.558632 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:01.894716 (+336084us) spinlock_profiling.cc:243] Waited 3.05 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:02.052628 (+157912us) raft_consensus.cc:1671] finished
0808 01:11:02.052649 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:02.052836 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":110,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":146,"spinlock_wait_cycles":7013248,"thread_start_us":90110,"threads_started":1,"wal-append.queue_time_us":88049}
I20260808 01:11:01.970126 4186 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:02.057906 4098 consensus_queue.cc:1035] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260808 01:11:02.058954 3954 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:02.059630 2202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 734) took 2171 ms. Trace:
I20260808 01:11:02.060657 2200 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260808 01:11:02.061095 2203 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 951) took 5275 ms (client timeout 1507 ms). Trace:
I20260808 01:11:02.062335 3999 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:02.063339 2201 raft_consensus.cc:1261] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260808 01:11:02.024883 3990 raft_consensus.cc:729] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
W20260808 01:11:02.059820 4031 consensus_meta.cc:287] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.533s user 0.005s sys 0.007s
I20260808 01:11:01.996908 3806 consensus_queue.cc:237] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.093545 4065 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:02.094185 3722 tablet_bootstrap.cc:492] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:11:02.093955 3990 consensus_queue.cc:260] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.095175 3990 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:11:02.097684 4127 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260808 01:11:02.100119 4161 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:02.082672 4102 raft_consensus.cc:2726] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:02.102382 4186 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:02.072150 4139 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:02.119036 4112 log.cc:924] Time spent T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 1.610s user 0.001s sys 0.012s
W20260808 01:11:02.121797 4196 log.cc:924] Time spent T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
W20260808 01:11:02.133293 4082 consensus_meta.cc:287] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.614s user 0.013s sys 0.000s
I20260808 01:11:02.133802 4139 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:02.135648 4170 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.158135 4170 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:02.187978 4140 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:02.219619 1329 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 161136us
W20260808 01:11:02.228396 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 222694us
W20260808 01:11:02.027875 1490 leader_election.cc:334] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.744s (SENT)
I20260808 01:11:02.231206 1490 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:02.232815 1438 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55036
W20260808 01:11:02.233656 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 205832us
I20260808 01:11:02.234481 1438 raft_consensus.cc:595] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Received request to transfer leadership
I20260808 01:11:02.235335 1438 raft_consensus.cc:603] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:11:02.237166 1517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1871) took 1465 ms. Trace:
W20260808 01:11:02.237192 1487 leader_election.cc:334] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.770s (SENT)
W20260808 01:11:02.243177 1487 leader_election.cc:334] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.244s (SENT)
W20260808 01:11:02.244148 1390 auto_rebalancer.cc:663] Could not move replica: Illegal state: not currently leader
W20260808 01:11:02.244692 1390 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: not currently leader
I20260808 01:11:02.244974 1410 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:02.237746 1517 rpcz_store.cc:276] 0808 01:11:00.771800 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:01.040831 (+269031us) service_pool.cc:227] Handling call
0808 01:11:01.261846 (+221015us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:01.296863 (+ 35017us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:01.296900 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:01.862809 (+565909us) spinlock_profiling.cc:243] Waited 4.18 ms on lock 0x7b2c00000238. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34b422e11 00007fe34b432442 00007fe34ca7ac11 00007fe34ca3f58a 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772
0808 01:11:01.863392 (+ 583us) log.cc:842] Serialized 26 byte log entry
0808 01:11:01.953045 (+ 89653us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:01.953095 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.953173 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:02.236237 (+283064us) raft_consensus.cc:1671] finished
0808 01:11:02.236304 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:02.236919 (+ 615us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34469,"raft.queue_time_us":40609,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":3810,"spinlock_wait_cycles":9616128,"thread_start_us":88438,"threads_started":1,"wal-append.queue_time_us":124498}
W20260808 01:11:02.251355 1490 leader_election.cc:334] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.848s (SENT)
I20260808 01:11:02.254827 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:02.263286 1446 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:02.255002 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:02.258109 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:02.259495 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:02.260686 1410 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:02.260934 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:02.266012 1448 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:11:02.266168 1442 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:02.269973 1597 raft_consensus.cc:1261] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260808 01:11:02.057793 1600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2469) took 2825 ms (client timeout 2802 ms). Trace:
W20260808 01:11:02.272264 1600 rpcz_store.cc:269] 0808 01:10:59.232659 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.232847 (+ 188us) service_pool.cc:227] Handling call
0808 01:11:02.057526 (+2824679us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b5400121454. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:02.057682 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2866853120}
I20260808 01:11:02.269974 1599 raft_consensus.cc:1261] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:11:02.272261 1598 raft_consensus.cc:1261] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:11:02.083766 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:02.070080 3689 raft_consensus.cc:729] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:02.095180 3722 tablet_bootstrap.cc:492] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: Bootstrap complete.
I20260808 01:11:02.292090 3748 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:02.293048 3748 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:02.294281 3722 ts_tablet_manager.cc:1397] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: Time spent bootstrapping tablet: real 2.679s user 0.049s sys 0.028s
I20260808 01:11:02.293421 3689 consensus_queue.cc:260] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.078595 3929 consensus_queue.cc:237] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:02.296489 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:02.297899 1604 raft_consensus.cc:2379] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 6: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:02.301369 3970 raft_consensus.cc:2726] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:02.302397 3970 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:02.088053 2203 rpcz_store.cc:269] 0808 01:10:56.785783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:56.815526 (+ 29743us) service_pool.cc:227] Handling call
0808 01:11:02.060718 (+5245192us) spinlock_profiling.cc:243] Waited 2.9 s on lock 0x7b54007e04d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:02.060980 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6663681280}
I20260808 01:11:02.312259 3840 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:02.312103 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:02.303799 2201 pending_rounds.cc:76] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd: Aborting all ops after (but not including) 5
I20260808 01:11:02.082373 2202 rpcz_store.cc:276] 0808 01:10:59.888691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.133936 (+245245us) service_pool.cc:227] Handling call
0808 01:11:02.059443 (+1925507us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54004c54d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:02.059565 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2805269376}
I20260808 01:11:02.303246 3970 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.326275 4199 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:02.332415 4148 raft_consensus.cc:2781] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:02.319949 2201 pending_rounds.cc:97] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260808 01:11:02.335367 4165 raft_consensus.cc:2781] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:11:02.336015 4140 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:02.328411 4199 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.336923 3689 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:02.338881 2201 raft_consensus.cc:2866] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:02.339457 4148 raft_consensus.cc:686] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:02.340565 4170 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:02.339000 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 87796.3us
I20260808 01:11:02.344467 4009 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:11:02.368633 2204 raft_consensus.cc:2396] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:11:02.364907 4170 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.372916 2209 raft_consensus.cc:1226] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 2. Current term is 4. Ops: []
I20260808 01:11:02.378468 4086 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:02.379171 4165 raft_consensus.cc:686] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:02.380307 4140 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:02.380664 1410 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:02.380609 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:02.381000 4165 consensus_queue.cc:237] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.381964 1448 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:11:02.382266 1438 raft_consensus.cc:1226] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:02.384377 4087 raft_consensus.cc:2726] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:02.365207 4148 consensus_queue.cc:237] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.303973 2171 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:49260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1006)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1016)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 742)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1002)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 591)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1003)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 582)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 577)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 584)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 598)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1018)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1007)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 588)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1021)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1005)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1013)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 583)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 585)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 576)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 580)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1019)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1020)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 590)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 595)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1009)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 589)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1022)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 748)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1024)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 596)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 597)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 586)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 587)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 599)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 594)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 746)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 735)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 737)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 738)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 740)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 741)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 743)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1004)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1010)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 744)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1023)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 745)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 747)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 578)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 579)
I20260808 01:11:02.313410 3840 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.389348 1410 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:02.390277 4199 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:02.392292 4098 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:02.394842 1602 raft_consensus.cc:1226] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:02.395642 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:02.403683 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:02.404238 1600 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:02.404778 1599 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
W20260808 01:11:02.405009 1518 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36560 (request call id 552) took 2158 ms (client timeout 1523 ms). Trace:
W20260808 01:11:02.405598 1490 leader_election.cc:334] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.948s (SENT)
W20260808 01:11:02.405515 1518 rpcz_store.cc:269] 0808 01:11:00.246649 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.246815 (+ 166us) service_pool.cc:227] Handling call
0808 01:11:02.404523 (+2157708us) spinlock_profiling.cc:243] Waited 99 ms on lock 0x7b54000dfa54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:02.404735 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":227764992}
I20260808 01:11:02.396133 4140 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.408351 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:02.409555 1438 raft_consensus.cc:2379] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:02.410488 4140 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:02.410738 2172 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:02.401799 4098 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.373759 4170 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:02.412911 4184 consensus_queue.cc:1035] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:02.416770 4098 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:02.416939 4014 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.418498 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:02.418660 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:02.423897 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:02.425235 1517 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
W20260808 01:11:02.426524 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 51036.5us
I20260808 01:11:02.427250 4170 raft_consensus.cc:2781] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:02.427920 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:02.428089 2172 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:02.428265 4033 raft_consensus.cc:3037] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:02.379578 4086 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.433773 3816 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:11:02.440798 1487 leader_election.cc:334] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.029s (SENT)
I20260808 01:11:02.441367 1487 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:02.430243 3970 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:02.445609 1410 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:02.429035 4170 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:02.448012 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:02.448405 4201 consensus_queue.cc:1046] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:11:02.451054 4138 raft_consensus.cc:2726] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:11:02.451785 4014 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:02.458850 3970 raft_consensus.cc:2726] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:02.454833 4170 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:02.481117 3992 raft_consensus.cc:3032] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:11:02.462057 4201 consensus_queue.cc:1225] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer 6b09dbefd24b4d52b385c4dff5697dc5 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260808 01:11:02.497520 4113 raft_consensus.cc:3032] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:02.499439 4134 raft_consensus.cc:3032] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:11:02.453735 4086 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:02.492826 4014 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:02.460902 4188 consensus_queue.cc:1035] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:11:02.504437 1600 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 7.
I20260808 01:11:02.512820 3992 raft_consensus.cc:729] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:02.521345 3990 raft_consensus.cc:2781] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 01:11:02.521591 4164 log.cc:924] Time spent T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 1.132s user 0.000s sys 0.003s
I20260808 01:11:02.521911 3990 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:02.522364 3992 consensus_queue.cc:260] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.461190 3929 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.053s
I20260808 01:11:02.540858 2208 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:11:02.530308 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:02.524662 1522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1866) took 1874 ms. Trace:
I20260808 01:11:02.531744 1446 raft_consensus.cc:1261] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260808 01:11:02.532351 4033 raft_consensus.cc:2726] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:02.535231 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:02.538050 1567 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:02.545861 3802 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:02.545945 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:02.546406 1600 raft_consensus.cc:1261] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:11:02.546519 3802 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.548105 1518 raft_consensus.cc:2396] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:11:02.548213 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:02.541874 1522 rpcz_store.cc:276] 0808 01:11:00.650120 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.710131 (+ 60011us) service_pool.cc:227] Handling call
0808 01:11:00.712347 (+ 2216us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:00.897817 (+185470us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:00.897862 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:00.930997 (+ 33135us) log.cc:842] Serialized 26 byte log entry
0808 01:11:01.274828 (+343831us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:01.274866 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.274935 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:02.515406 (+1240471us) spinlock_profiling.cc:243] Waited 56.3 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:02.524102 (+ 8696us) raft_consensus.cc:1671] finished
0808 01:11:02.524122 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:02.524476 (+ 354us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45191,"raft.queue_time_us":127461,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":76757,"spinlock_wait_cycles":129565440,"thread_start_us":195940,"threads_started":1,"wal-append.queue_time_us":308932}
W20260808 01:11:02.550437 1565 leader_election.cc:334] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.805s (SENT)
I20260808 01:11:02.551043 1565 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:02.518072 4113 raft_consensus.cc:729] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:02.518186 4134 raft_consensus.cc:729] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:02.522393 3990 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:02.557971 4041 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:02.541857 3992 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:02.565363 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:02.587949 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 50118.8us
I20260808 01:11:02.559105 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:02.559386 4014 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:02.644121 3954 consensus_meta.cc:287] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.561s user 0.004s sys 0.019s
I20260808 01:11:02.630244 3689 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:02.645505 3689 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.649032 3689 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:02.656620 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:02.630390 3911 consensus_meta.cc:287] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.519s user 0.014s sys 0.000s
I20260808 01:11:02.631476 3929 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:02.667788 4041 raft_consensus.cc:2781] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:02.691545 4067 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:02.692250 4067 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:02.694773 4126 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:02.696476 4036 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:02.697283 4036 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.676532 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:02.700167 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 77808.1us
I20260808 01:11:02.700632 1444 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
W20260808 01:11:02.688107 4169 log.cc:924] Time spent T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20260808 01:11:02.701649 2201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 967) took 5695 ms. Trace:
I20260808 01:11:02.702188 2201 rpcz_store.cc:276] 0808 01:10:57.006608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.241672 (+3235064us) service_pool.cc:227] Handling call
0808 01:11:02.062982 (+1821310us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:02.364945 (+301963us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:02.701383 (+336438us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b54007e04d0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:02.701563 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4145109376}
I20260808 01:11:02.702364 2172 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:02.685081 1568 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:02.703913 4134 consensus_queue.cc:260] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:02.705188 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 161214us
I20260808 01:11:02.705364 4057 raft_consensus.cc:2726] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:02.705533 4134 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:02.691668 4041 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:02.706102 1568 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:02.689024 4185 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:02.706969 4185 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.708463 3954 raft_consensus.cc:2781] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:02.708938 3954 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:02.709334 3954 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:02.709744 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:02.707549 1568 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:02.704043 3689 raft_consensus.cc:2781] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:02.712604 3689 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:02.713044 3689 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:02.712421 4113 consensus_queue.cc:260] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:02.713635 4067 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:02.706233 4041 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:02.709919 4036 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:02.681600 1438 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:02.689904 3861 raft_consensus.cc:2726] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:02.704605 2209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 949) took 5919 ms. Trace:
I20260808 01:11:02.706998 1447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 646) took 1142 ms. Trace:
W20260808 01:11:02.720192 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 243990us
I20260808 01:11:02.717811 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:02.726675 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 281188us
I20260808 01:11:02.720834 1447 rpcz_store.cc:276] 0808 01:11:01.564139 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:01.564279 (+ 140us) service_pool.cc:227] Handling call
0808 01:11:01.600675 (+ 36396us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:01.636675 (+ 36000us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:02.706552 (+1069877us) spinlock_profiling.cc:243] Waited 35.2 ms on lock 0x7b540002dad4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:02.706770 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":80951552}
I20260808 01:11:02.720253 2209 rpcz_store.cc:276] 0808 01:10:56.785268 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.138654 (+3353386us) service_pool.cc:227] Handling call
0808 01:11:01.347614 (+1208960us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x7b5800005198. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adf851 00007fe349adf702 00007fe3529cfb09 00007fe35190b730 00007fe3519d179b 00007fe3519d1821 00007fe3519952b0 00007fe3519807f9 00007fe3519802a2 00007fe34b066684
0808 01:11:02.372405 (+1024791us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:02.373446 (+ 1041us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:02.704408 (+330962us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54007e04d0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:02.704553 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2357510400}
I20260808 01:11:02.719462 4115 raft_consensus.cc:2781] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:02.735049 3722 raft_consensus.cc:348] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:02.745577 1410 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:02.717727 3911 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.753613 1568 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:02.766242 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:02.722992 4067 consensus_queue.cc:1035] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.015s
I20260808 01:11:02.766541 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:02.768075 1440 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:02.766592 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:02.738369 4115 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:02.739779 3772 raft_consensus.cc:2726] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:02.750401 3722 raft_consensus.cc:729] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Initialized, Role: LEARNER
I20260808 01:11:02.784427 1568 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:02.749085 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:02.792171 1565 leader_election.cc:334] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.043s (SENT)
I20260808 01:11:02.782650 4057 raft_consensus.cc:2726] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 01:11:02.797957 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 194288us
I20260808 01:11:02.744899 3929 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:02.803689 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:02.800588 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:02.804984 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:02.805593 2202 raft_consensus.cc:2396] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260808 01:11:02.806218 2202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1006) took 1238 ms. Trace:
I20260808 01:11:02.806638 2202 rpcz_store.cc:276] 0808 01:11:01.567544 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.734102 (+1166558us) service_pool.cc:227] Handling call
0808 01:11:02.806137 (+ 72035us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:02.808327 1439 raft_consensus.cc:2396] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260808 01:11:02.809850 2205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1017) took 1182 ms. Trace:
I20260808 01:11:02.810395 2205 rpcz_store.cc:276] 0808 01:11:01.627301 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.748542 (+1121241us) service_pool.cc:227] Handling call
0808 01:11:02.809732 (+ 61190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:02.814008 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:02.816514 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:02.817561 2209 raft_consensus.cc:2379] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 6: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:02.818101 2209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1003) took 1251 ms. Trace:
I20260808 01:11:02.818346 4014 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:02.818533 2209 rpcz_store.cc:276] 0808 01:11:01.566182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.816130 (+1249948us) service_pool.cc:227] Handling call
0808 01:11:02.818018 (+ 1888us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:02.823215 3748 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:02.825785 3954 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:02.826545 3802 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:02.829113 2208 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 733) took 2957 ms (client timeout 1982 ms). Trace:
W20260808 01:11:02.829558 2208 rpcz_store.cc:269] 0808 01:10:59.888244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:59.979339 (+ 91095us) service_pool.cc:227] Handling call
0808 01:11:02.828880 (+2849541us) spinlock_profiling.cc:243] Waited 2.23 s on lock 0x7b54004cf254. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:02.829019 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5128615680}
I20260808 01:11:02.830212 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:02.830731 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:02.831079 1517 raft_consensus.cc:2454] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:11:02.830200 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:02.833330 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:02.833930 1567 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:02.834748 4170 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.831498 2208 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:02.839843 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:02.840076 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 94616.8us
I20260808 01:11:02.836890 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:02.843250 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 57304.4us
I20260808 01:11:02.844383 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:02.846693 2202 raft_consensus.cc:2396] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260808 01:11:02.792892 4087 raft_consensus.cc:2726] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:02.848524 3722 consensus_queue.cc:260] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:02.851354 2200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 555) took 3598 ms. Trace:
I20260808 01:11:02.851878 2200 rpcz_store.cc:276] 0808 01:10:59.252499 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.504062 (+1251563us) service_pool.cc:227] Handling call
0808 01:11:02.060303 (+1556241us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:02.082863 (+ 22560us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:02.850843 (+767980us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b54004c54d0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:02.851125 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3577817984}
I20260808 01:11:02.853375 3722 ts_tablet_manager.cc:1428] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: Time spent starting tablet: real 0.558s user 0.006s sys 0.012s
I20260808 01:11:02.853457 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:02.854728 2200 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:02.855145 3954 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:02.855403 2200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1007) took 1287 ms. Trace:
I20260808 01:11:02.857138 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:02.859308 3861 raft_consensus.cc:2781] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 01:11:02.859766 4224 log.cc:924] Time spent T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.131s user 0.000s sys 0.001s
I20260808 01:11:02.859907 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:02.871174 2204 raft_consensus.cc:2362] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3. Re-sending same reply.
I20260808 01:11:02.860551 3911 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:02.861404 1516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2079) took 1313 ms. Trace:
W20260808 01:11:02.861701 4220 log.cc:924] Time spent T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260808 01:11:02.861532 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:02.862390 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:02.862497 4040 raft_consensus.cc:2781] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:02.857661 2202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 582) took 1002 ms. Trace:
I20260808 01:11:02.882216 2202 rpcz_store.cc:276] 0808 01:11:01.854747 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.807520 (+952773us) service_pool.cc:227] Handling call
0808 01:11:02.857596 (+ 50076us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:02.885334 1445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55042 (request call id 2330) took 1103 ms. Trace:
I20260808 01:11:02.885864 1445 rpcz_store.cc:276] 0808 01:11:01.781382 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:01.781553 (+ 171us) service_pool.cc:227] Handling call
0808 01:11:01.804519 (+ 22966us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:01.807134 (+ 2615us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:01.807176 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:01.807890 (+ 714us) log.cc:842] Serialized 26 byte log entry
0808 01:11:02.420461 (+612571us) spinlock_profiling.cc:243] Waited 2.52 ms on lock 0x7b1800408e20. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349af9345 00007fe352930a39 00007fe349b168bb 00007fe349b14cbf 00007fe349b16e22 00007fe34c6c05bb 00007fe34c6c6063 00007fe34c6c6411 00007fe34ca135cf
0808 01:11:02.712467 (+292006us) raft_consensus.cc:1631] Marking committed up to 2
0808 01:11:02.714770 (+ 2303us) log.cc:842] Serialized 14 byte log entry
0808 01:11:02.715049 (+ 279us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:02.727138 (+ 12089us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:02.884712 (+157574us) raft_consensus.cc:1671] finished
0808 01:11:02.884867 (+ 155us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:02.885077 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4349,"prepare.queue_time_us":1650,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":59,"raft.queue_time_us":7859,"raft.run_cpu_time_us":464,"raft.run_wall_time_us":26951,"spinlock_wait_cycles":5789056,"thread_start_us":250249,"threads_started":1,"wal-append.queue_time_us":906556}
I20260808 01:11:02.872375 1516 rpcz_store.cc:276] 0808 01:11:01.548390 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:01.548564 (+ 174us) service_pool.cc:227] Handling call
0808 01:11:01.616919 (+ 68355us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:01.796182 (+179263us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:02.403196 (+607014us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b540035d4d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:02.861001 (+457805us) spinlock_profiling.cc:243] Waited 48.8 ms on lock 0x7b540035d4d0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:02.861219 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":512146560}
I20260808 01:11:02.891366 2208 raft_consensus.cc:2396] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:11:02.796592 4115 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:11:02.853909 3722 ts_tablet_manager.cc:1460] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: Tablet startup took 559ms
I20260808 01:11:02.870569 2200 rpcz_store.cc:276] 0808 01:11:01.567839 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.853057 (+1285218us) service_pool.cc:227] Handling call
0808 01:11:02.855312 (+ 2255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:02.871830 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 742) took 1336 ms. Trace:
I20260808 01:11:02.900368 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:02.841257 2205 raft_consensus.cc:2354] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6.
I20260808 01:11:02.863559 2209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 584) took 1008 ms. Trace:
I20260808 01:11:02.844135 4067 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:02.894579 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1018) took 1240 ms. Trace:
W20260808 01:11:02.896927 4168 log.cc:924] Time spent T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 1.340s user 0.004s sys 0.012s
I20260808 01:11:02.899009 2206 raft_consensus.cc:2396] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 01:11:02.901765 2209 rpcz_store.cc:276] 0808 01:11:01.855066 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.819526 (+964460us) service_pool.cc:227] Handling call
0808 01:11:02.863491 (+ 43965us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:02.890489 1410 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:02.907341 2208 rpcz_store.cc:276] 0808 01:11:01.654537 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.855944 (+1201407us) service_pool.cc:227] Handling call
0808 01:11:02.894496 (+ 38552us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:02.872673 3861 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:02.900799 2204 rpcz_store.cc:276] 0808 01:11:01.535892 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.859540 (+1323648us) service_pool.cc:227] Handling call
0808 01:11:02.871765 (+ 12225us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:02.914633 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 56526.5us
I20260808 01:11:02.918988 4188 consensus_queue.cc:1035] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:11:02.882135 4040 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:11:02.924628 2206 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1011) took 1355 ms (client timeout 1724 ms). Trace:
W20260808 01:11:02.939638 2206 rpcz_store.cc:269] 0808 01:11:01.568889 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.861955 (+1293066us) service_pool.cc:227] Handling call
0808 01:11:02.924559 (+ 62604us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:02.915938 3861 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:02.950321 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:02.950912 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:02.952175 2202 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
W20260808 01:11:02.954139 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 150035us
I20260808 01:11:02.954319 2202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 588) took 1098 ms. Trace:
I20260808 01:11:02.955315 2200 raft_consensus.cc:2379] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 4: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:02.942317 4040 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:02.955504 2202 rpcz_store.cc:276] 0808 01:11:01.855952 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.906533 (+1050581us) service_pool.cc:227] Handling call
0808 01:11:02.954236 (+ 47703us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:02.957132 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:02.955084 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:02.959223 2200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1021) took 1278 ms. Trace:
I20260808 01:11:02.959681 2200 rpcz_store.cc:276] 0808 01:11:01.681280 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.912891 (+1231611us) service_pool.cc:227] Handling call
0808 01:11:02.959162 (+ 46271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:02.960371 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:02.916738 4067 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:02.962483 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:02.948846 3929 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:02.958294 2209 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:02.965958 2209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1005) took 1399 ms. Trace:
I20260808 01:11:02.963255 2208 raft_consensus.cc:2379] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 3: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:02.950827 2205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 577) took 1094 ms. Trace:
I20260808 01:11:02.953168 4087 raft_consensus.cc:2726] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:02.966293 2209 rpcz_store.cc:276] 0808 01:11:01.566672 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.925426 (+1358754us) service_pool.cc:227] Handling call
0808 01:11:02.965893 (+ 40467us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:02.967391 2201 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:02.942617 3970 consensus_queue.cc:1035] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
W20260808 01:11:02.966856 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 86445.6us
I20260808 01:11:02.966908 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 583) took 1111 ms. Trace:
I20260808 01:11:02.967072 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1013) took 1397 ms. Trace:
I20260808 01:11:02.972122 2208 rpcz_store.cc:276] 0808 01:11:01.854909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.944869 (+1089960us) service_pool.cc:227] Handling call
0808 01:11:02.966806 (+ 21937us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:02.972183 2206 raft_consensus.cc:2354] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4.
I20260808 01:11:02.972301 2204 rpcz_store.cc:276] 0808 01:11:01.569404 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.928237 (+1358833us) service_pool.cc:227] Handling call
0808 01:11:02.967015 (+ 38778us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:02.942414 4041 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:02.960428 3840 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:02.969272 1565 leader_election.cc:334] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.189s (SENT)
I20260808 01:11:02.975085 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:02.980656 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 585) took 1121 ms. Trace:
I20260808 01:11:02.980991 2206 rpcz_store.cc:276] 0808 01:11:01.855226 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.947051 (+1091825us) service_pool.cc:227] Handling call
0808 01:11:02.976714 (+ 29663us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:02.900315 3722 ts_tablet_manager.cc:1462] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: Trace:
0808 01:10:44.359919 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:44.360033 (+ 114us) service_pool.cc:227] Handling call
0808 01:10:44.451796 (+ 91763us) ts_tablet_manager.cc:928] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: Initiating tablet copy from peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
0808 01:10:45.272935 (+821139us) spinlock_profiling.cc:243] Waited 1.45 ms on lock 0x7fe349ba2290. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349b3f641 00007fe349b15dde 00007fe349b13534 00007fe351930691 00007fe351936ef8 00007fe3519cc81e 00007fe3519fda8e 00007fe3519e35ad 00007fe3519e3559 00007fe3519e34e9
0808 01:10:47.584071 (+2311136us) tablet_copy_client.cc:361] Tablet copy session begun
0808 01:10:48.716785 (+1132714us) tablet_metadata.cc:641] Metadata flushed
0808 01:10:48.732733 (+ 15948us) tablet_copy_client.cc:490] Wrote new tablet metadata
0808 01:10:49.503137 (+770404us) tablet_copy_client.cc:411] Wrote new consensus metadata
0808 01:10:49.595518 (+ 92381us) tablet_replica.cc:205] Creating consensus instance
0808 01:10:49.778206 (+182688us) ts_tablet_manager.cc:983] Replica registered.
0808 01:10:49.868821 (+ 90615us) inbound_call.cc:173] Queueing success response
0808 01:10:59.598620 (+9729799us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0808 01:11:02.295224 (+2696604us) ts_tablet_manager.cc:1429] Starting tablet replica
0808 01:11:02.329056 (+ 33832us) tablet_replica.cc:259] Starting instrumentation
0808 01:11:02.369466 (+ 40410us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":244,"mutex_wait_us":28046,"raft.queue_time_us":132,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":183,"spinlock_wait_cycles":3326336,"tablet-copy.queue_time_us":90645,"tablet-download-pool.queue_time_us":303,"tablet-download-pool.run_cpu_time_us":3420,"tablet-download-pool.run_wall_time_us":7695052,"thread_start_us":108549,"threads_started":2}
I20260808 01:11:02.984170 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:02.984956 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:02.987499 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:02.989272 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:02.989447 1438 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:02.990094 1600 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:02.980384 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:02.991955 2200 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:02.992322 3990 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:02.993484 3795 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:02.994971 3990 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:02.999958 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:03.006233 4170 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:03.007061 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.008374 4131 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:03.008737 2207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 727) took 3633 ms. Trace:
I20260808 01:11:03.011368 1519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1858) took 2386 ms. Trace:
I20260808 01:11:03.013345 4041 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:03.013490 4067 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:03.019580 2200 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:03.018103 2202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 576) took 1165 ms. Trace:
I20260808 01:11:03.020164 2200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 580) took 1165 ms. Trace:
I20260808 01:11:03.021466 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.020493 2202 rpcz_store.cc:276] 0808 01:11:01.853124 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.964230 (+1111106us) service_pool.cc:227] Handling call
0808 01:11:03.018029 (+ 53799us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.022050 2200 rpcz_store.cc:276] 0808 01:11:01.854320 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.963988 (+1109668us) service_pool.cc:227] Handling call
0808 01:11:03.020105 (+ 56117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.023110 1410 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:03.023754 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:02.968369 2205 rpcz_store.cc:276] 0808 01:11:01.853332 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.811346 (+958014us) service_pool.cc:227] Handling call
0808 01:11:02.945895 (+134549us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.025010 2200 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:03.029805 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:03.032544 2209 raft_consensus.cc:2379] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 4: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:03.033078 2209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1019) took 1352 ms. Trace:
I20260808 01:11:03.033465 2209 rpcz_store.cc:276] 0808 01:11:01.680499 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.979095 (+1298596us) service_pool.cc:227] Handling call
0808 01:11:03.032995 (+ 53900us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.038209 4184 raft_consensus.cc:2726] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:11:03.038739 4040 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:03.031371 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.018190 4131 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:03.016640 2207 rpcz_store.cc:276] 0808 01:10:59.375674 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:01.310821 (+1935147us) service_pool.cc:227] Handling call
0808 01:11:01.311173 (+ 352us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:01.311407 (+ 234us) raft_consensus.cc:1520] Early marking committed up to index 5
0808 01:11:01.322251 (+ 10844us) log.cc:842] Serialized 14 byte log entry
0808 01:11:02.850448 (+1528197us) spinlock_profiling.cc:243] Waited 1.71 ms on lock 0x7b180036e380. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349af9345 00007fe352930a39 00007fe349b168bb 00007fe349b14cbf 00007fe349b16e22 00007fe34c6c05bb 00007fe34c6c6063 00007fe34c6c657b 00007fe34ca4988a
0808 01:11:03.008106 (+157658us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:03.008179 (+ 73us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:03.008292 (+ 113us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.008376 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:03.008547 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":6522,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":27131,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":201,"spinlock_wait_cycles":3934080,"thread_start_us":117320,"threads_started":1}
W20260808 01:11:03.042945 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 73696.1us
W20260808 01:11:03.044100 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 82652.2us
I20260808 01:11:02.979362 4140 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:03.047986 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.053267 1568 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:03.039592 4184 consensus_queue.cc:1035] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:11:03.055190 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:03.057860 1568 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:03.055421 2208 raft_consensus.cc:2396] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:11:03.055536 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 590) took 1047 ms. Trace:
I20260808 01:11:03.059957 2204 rpcz_store.cc:276] 0808 01:11:02.008036 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.979238 (+971202us) service_pool.cc:227] Handling call
0808 01:11:03.055409 (+ 76171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.060822 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1020) took 1380 ms. Trace:
I20260808 01:11:03.061185 2208 rpcz_store.cc:276] 0808 01:11:01.680909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.973870 (+1292961us) service_pool.cc:227] Handling call
0808 01:11:03.060764 (+ 86894us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.062768 1603 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:03.063961 2206 raft_consensus.cc:2379] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 7: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:03.064087 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.064850 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 595) took 1019 ms. Trace:
I20260808 01:11:03.065198 2206 rpcz_store.cc:276] 0808 01:11:02.045537 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.981520 (+935983us) service_pool.cc:227] Handling call
0808 01:11:03.064793 (+ 83273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.065604 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.019255 3689 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:03.073555 1599 raft_consensus.cc:1261] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260808 01:11:03.076123 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.079481 1597 raft_consensus.cc:1226] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:03.080394 4131 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:00.933071 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:11:03.084124 4144 raft_consensus.cc:2726] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:03.085126 3748 raft_consensus.cc:2781] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:03.085753 3748 raft_consensus.cc:686] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:03.085209 1340 catalog_manager.cc:5495] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 3 to 4, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 4 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:11:03.086707 3748 consensus_queue.cc:237] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:03.063542 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.091090 4185 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:03.091717 2205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1022) took 1410 ms. Trace:
I20260808 01:11:03.092119 2205 rpcz_store.cc:276] 0808 01:11:01.681584 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.048049 (+1366465us) service_pool.cc:227] Handling call
0808 01:11:03.091656 (+ 43607us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.074128 1599 pending_rounds.cc:76] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78: Aborting all ops after (but not including) 2
I20260808 01:11:03.093077 1599 pending_rounds.cc:97] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260808 01:11:03.093351 1599 raft_consensus.cc:2866] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:03.093200 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.093886 2205 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:03.095006 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.096935 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:11:03.095028 2201 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1002) took 1529 ms (client timeout 1954 ms). Trace:
W20260808 01:11:03.100680 2201 rpcz_store.cc:269] 0808 01:11:01.565937 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:02.703405 (+1137468us) service_pool.cc:227] Handling call
0808 01:11:03.094817 (+391412us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b54005d0554. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:03.094959 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":543472000}
I20260808 01:11:03.015170 1519 rpcz_store.cc:276] 0808 01:11:00.624645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.624845 (+ 200us) service_pool.cc:227] Handling call
0808 01:11:00.638758 (+ 13913us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:00.639121 (+ 363us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:00.639147 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:00.661289 (+ 22142us) log.cc:842] Serialized 26 byte log entry
0808 01:11:01.011377 (+350088us) spinlock_profiling.cc:243] Waited 6.63 ms on lock 0x7b1800466180. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349af9345 00007fe352930a39 00007fe349b168bb 00007fe349b14cbf 00007fe349b16e22 00007fe34c6c05bb 00007fe34c6c6063 00007fe34c6c6411 00007fe34ca135cf
0808 01:11:01.542280 (+530903us) raft_consensus.cc:1631] Marking committed up to 2
0808 01:11:01.643730 (+101450us) log.cc:842] Serialized 14 byte log entry
0808 01:11:01.658044 (+ 14314us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:01.658161 (+ 117us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:02.895680 (+1237519us) spinlock_profiling.cc:243] Waited 6.52 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:03.010912 (+115232us) raft_consensus.cc:1671] finished
0808 01:11:03.010973 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:03.011188 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13920,"prepare.queue_time_us":128009,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":27619,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":18557,"spinlock_wait_cycles":30237696,"thread_start_us":598575,"threads_started":2,"wal-append.queue_time_us":894827}
I20260808 01:11:03.108217 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:11:03.109220 4219 log.cc:924] Time spent T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.302s user 0.001s sys 0.001s
I20260808 01:11:03.109565 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:03.109999 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:03.110760 1444 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 9.
I20260808 01:11:03.109189 1443 raft_consensus.cc:2379] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:03.114887 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.116328 2204 raft_consensus.cc:2379] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 4: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:03.112121 2202 raft_consensus.cc:2454] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 2.
I20260808 01:11:03.118363 2202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1009) took 1549 ms. Trace:
I20260808 01:11:03.118840 2202 rpcz_store.cc:276] 0808 01:11:01.568369 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.031557 (+1463188us) service_pool.cc:227] Handling call
0808 01:11:03.118169 (+ 86612us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.098742 2200 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:11:03.119506 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.120362 2200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 748) took 1315 ms. Trace:
I20260808 01:11:03.120920 2200 rpcz_store.cc:276] 0808 01:11:01.805329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.058679 (+1253350us) service_pool.cc:227] Handling call
0808 01:11:03.120305 (+ 61626us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37248}
I20260808 01:11:03.125373 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1024) took 1443 ms. Trace:
I20260808 01:11:03.125946 2204 rpcz_store.cc:276] 0808 01:11:01.682221 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.060625 (+1378404us) service_pool.cc:227] Handling call
0808 01:11:03.125289 (+ 64664us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.121263 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.132396 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:03.123878 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.097640 1438 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:03.134407 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:03.095727 2205 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:11:03.136493 1523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2085) took 1552 ms. Trace:
I20260808 01:11:03.132398 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:03.147598 2209 raft_consensus.cc:2396] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260808 01:11:03.063112 4165 consensus_queue.cc:1046] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260808 01:11:03.071300 4234 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:03.144248 4099 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:03.145540 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:03.110780 3861 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:03.148350 2208 raft_consensus.cc:2379] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 4: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:03.059697 4140 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:03.155542 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 596) took 1081 ms. Trace:
I20260808 01:11:03.155977 2207 raft_consensus.cc:2396] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:11:03.156019 2208 rpcz_store.cc:276] 0808 01:11:02.074307 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.061902 (+987595us) service_pool.cc:227] Handling call
0808 01:11:03.155479 (+ 93577us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.148986 1447 raft_consensus.cc:2379] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 4: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:03.151039 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.146729 1523 rpcz_store.cc:276] 0808 01:11:01.569645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:01.608315 (+ 38670us) service_pool.cc:227] Handling call
0808 01:11:01.608594 (+ 279us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:01.655573 (+ 46979us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:01.655635 (+ 62us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:01.656147 (+ 512us) log.cc:842] Serialized 26 byte log entry
0808 01:11:02.407514 (+751367us) spinlock_profiling.cc:243] Waited 8.32 ms on lock 0x7b1800394b40. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349af9345 00007fe352930a39 00007fe349b168bb 00007fe349b14cbf 00007fe349b16e22 00007fe34c6c05bb 00007fe34c6c6063 00007fe34c6c6411 00007fe34ca135cf
0808 01:11:02.645360 (+237846us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:02.645847 (+ 487us) log.cc:842] Serialized 14 byte log entry
0808 01:11:02.646125 (+ 278us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:02.646222 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:03.108442 (+462220us) spinlock_profiling.cc:243] Waited 33.5 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:03.111577 (+ 3135us) raft_consensus.cc:1671] finished
0808 01:11:03.111598 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:03.121953 (+ 10355us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":765,"prepare.queue_time_us":192,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":48256,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":18257,"spinlock_wait_cycles":96199424,"thread_start_us":203304,"threads_started":2,"wal-append.queue_time_us":1081909}
I20260808 01:11:03.158938 4165 consensus_queue.cc:1225] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer 6b09dbefd24b4d52b385c4dff5697dc5 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 01:11:03.153466 1601 raft_consensus.cc:2454] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:03.159729 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:03.160018 4165 raft_consensus.cc:3032] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:11:03.160414 4165 raft_consensus.cc:729] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:03.160638 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:03.161540 1601 raft_consensus.cc:2396] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260808 01:11:03.161257 4165 consensus_queue.cc:260] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:03.162436 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:03.158428 2204 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:11:03.159528 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:03.153720 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:03.163234 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 594) took 1144 ms. Trace:
I20260808 01:11:03.163771 1445 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:03.163751 2204 rpcz_store.cc:276] 0808 01:11:02.018509 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.136253 (+1117744us) service_pool.cc:227] Handling call
0808 01:11:03.163145 (+ 26892us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.153235 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:03.165153 2200 raft_consensus.cc:2379] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 4: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:03.165748 2209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 589) took 1309 ms. Trace:
I20260808 01:11:03.166146 2209 rpcz_store.cc:276] 0808 01:11:01.856200 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.034265 (+1178065us) service_pool.cc:227] Handling call
0808 01:11:03.165691 (+131426us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:03.166800 1565 leader_election.cc:334] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.174s (SENT)
I20260808 01:11:03.167093 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.167382 1600 raft_consensus.cc:2396] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:11:03.168397 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:03.169371 2203 raft_consensus.cc:2396] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260808 01:11:03.168866 4234 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:03.169811 2203 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1016) took 1556 ms (client timeout 1770 ms). Trace:
I20260808 01:11:03.170063 4140 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:03.170212 2203 rpcz_store.cc:269] 0808 01:11:01.613101 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.167958 (+1554857us) service_pool.cc:227] Handling call
0808 01:11:03.169760 (+ 1802us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.171265 4234 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:03.167789 2209 raft_consensus.cc:2454] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:03.172287 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.167279 1565 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:03.174240 1599 raft_consensus.cc:2354] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4.
I20260808 01:11:03.175644 1602 raft_consensus.cc:2354] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 4. Current term is 5.
I20260808 01:11:03.176299 2202 raft_consensus.cc:2379] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:03.178419 2202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 599) took 1103 ms. Trace:
I20260808 01:11:03.178679 2202 rpcz_store.cc:276] 0808 01:11:02.075374 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.119675 (+1044301us) service_pool.cc:227] Handling call
0808 01:11:03.178372 (+ 58697us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.182812 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:03.150879 2205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 586) took 1295 ms. Trace:
I20260808 01:11:03.171528 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.186790 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:03.186941 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:03.187497 2204 raft_consensus.cc:2379] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 4: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:03.179311 2206 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 1.
I20260808 01:11:03.188364 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 597) took 1113 ms. Trace:
I20260808 01:11:03.188623 2206 rpcz_store.cc:276] 0808 01:11:02.074623 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.071080 (+996457us) service_pool.cc:227] Handling call
0808 01:11:03.188314 (+117234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.189455 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.192112 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.197065 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.197582 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:03.199961 2209 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:03.196864 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:03.200554 1448 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:03.200320 2208 raft_consensus.cc:2379] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 4: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:03.198278 1602 raft_consensus.cc:2454] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:11:03.201638 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 746) took 1400 ms. Trace:
I20260808 01:11:03.203225 1568 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:03.203138 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.204975 1600 raft_consensus.cc:1261] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:11:03.205361 2206 raft_consensus.cc:2396] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:11:03.209537 3861 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:03.211473 4113 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:03.186517 2205 rpcz_store.cc:276] 0808 01:11:01.855438 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.094694 (+1239256us) service_pool.cc:227] Handling call
0808 01:11:03.149125 (+ 54431us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.213358 2205 raft_consensus.cc:1261] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:11:03.213999 2205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 740) took 2408 ms. Trace:
I20260808 01:11:03.214329 2205 rpcz_store.cc:276] 0808 01:11:00.805691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.212601 (+2406910us) service_pool.cc:227] Handling call
0808 01:11:03.212971 (+ 370us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:03.213707 (+ 736us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.213894 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.216046 1518 raft_consensus.cc:1226] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260808 01:11:03.217047 4098 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:11:03.217464 4098 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:03.219928 2208 rpcz_store.cc:276] 0808 01:11:01.801450 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.174888 (+1373438us) service_pool.cc:227] Handling call
0808 01:11:03.201555 (+ 26667us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.220597 1604 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 1.
I20260808 01:11:03.222561 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:03.224984 2202 raft_consensus.cc:1226] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:03.225473 1522 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:03.226774 2207 raft_consensus.cc:2396] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:11:03.217759 4115 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:03.227815 1596 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:11:03.227967 4072 raft_consensus.cc:2726] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 01:11:03.229002 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 62284us
I20260808 01:11:03.229431 4115 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:03.230541 1594 raft_consensus.cc:1226] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260808 01:11:03.230880 3689 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:03.231526 1408 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
W20260808 01:11:03.232390 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 57119.2us
I20260808 01:11:03.234239 1408 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:03.235163 3801 raft_consensus.cc:2781] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:03.235715 3801 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:03.236209 3801 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:03.236421 2171 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:03.217800 4098 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:11:03.234259 3772 raft_consensus.cc:2781] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:03.237874 3772 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:03.237936 2171 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:03.238482 3772 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:03.240744 1338 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55752 (request call id 306) took 10541 ms. Trace:
W20260808 01:11:03.240978 4217 log.cc:924] Time spent T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.409s user 0.003s sys 0.002s
I20260808 01:11:03.242203 4233 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:03.242530 1521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1864) took 2593 ms. Trace:
I20260808 01:11:03.242879 4233 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:03.242456 4009 consensus_queue.cc:1035] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260808 01:11:03.243005 1521 rpcz_store.cc:276] 0808 01:11:00.648803 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:00.701810 (+ 53007us) service_pool.cc:227] Handling call
0808 01:11:00.832363 (+130553us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:00.832692 (+ 329us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:00.832729 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:02.382282 (+1549553us) spinlock_profiling.cc:243] Waited 9.08 ms on lock 0x7b2c00000238. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34b422e11 00007fe34b432442 00007fe34ca7ac11 00007fe34ca3f58a 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772
0808 01:11:02.383138 (+ 856us) log.cc:842] Serialized 26 byte log entry
0808 01:11:02.673431 (+290293us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:02.673474 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:02.679241 (+ 5767us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:03.190260 (+511019us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:03.242189 (+ 51929us) raft_consensus.cc:1671] finished
0808 01:11:03.242211 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:03.242372 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47231616,"thread_start_us":289846,"threads_started":1,"wal-append.queue_time_us":367393}
I20260808 01:11:03.218430 4040 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:03.246064 2200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 735) took 3357 ms. Trace:
I20260808 01:11:03.246475 2200 rpcz_store.cc:276] 0808 01:10:59.889006 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.177818 (+3288812us) service_pool.cc:227] Handling call
0808 01:11:03.205965 (+ 28147us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:03.221763 (+ 15798us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:03.221810 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:03.221866 (+ 56us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:03.245796 (+ 23930us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.245873 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:03.245988 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.247720 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.248749 4233 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:03.249235 2200 raft_consensus.cc:3037] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 2 LEARNER]: Advancing to term 4
I20260808 01:11:03.250269 2202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 737) took 3360 ms. Trace:
I20260808 01:11:03.250699 2202 rpcz_store.cc:276] 0808 01:10:59.889583 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.186150 (+3296567us) service_pool.cc:227] Handling call
0808 01:11:03.222427 (+ 36277us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:03.248897 (+ 26470us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.250149 (+ 1252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.249533 4233 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:03.196151 2203 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:03.252697 4233 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:03.253196 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:03.253665 2203 raft_consensus.cc:1261] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:11:03.254010 2209 raft_consensus.cc:2379] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 6: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:03.254268 2203 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1023) took 1572 ms. Trace:
I20260808 01:11:03.241176 1338 rpcz_store.cc:276] 0808 01:10:52.698844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:52.700063 (+ 1219us) service_pool.cc:227] Handling call
0808 01:11:03.240470 (+10540407us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 01:10:57.637937 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:10:57.842321 (+204384us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:10:57.847987 (+ 5666us) write_op.cc:435] Acquired schema lock
0808 01:10:57.848015 (+ 28us) tablet.cc:660] Decoding operations
0808 01:10:58.094100 (+246085us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:10:58.094167 (+ 67us) write_op.cc:641] Partition lock acquired for write op
0808 01:10:58.094194 (+ 27us) tablet.cc:683] Acquiring locks for 6 operations
0808 01:10:58.094716 (+ 522us) tablet.cc:699] Row locks acquired
0808 01:10:58.094735 (+ 19us) write_op.cc:260] PREPARE: finished
0808 01:10:58.094897 (+ 162us) write_op.cc:270] Start()
0808 01:10:58.104122 (+ 9225us) write_op.cc:276] Timestamp: P: 1786151458094845 usec, L: 0
0808 01:10:58.104148 (+ 26us) op_driver.cc:331] REPLICATION: starting
0808 01:10:58.182839 (+ 78691us) log.cc:842] Serialized 2583 byte log entry
0808 01:10:59.275586 (+1092747us) op_driver.cc:444] REPLICATION: finished
0808 01:10:59.942990 (+667404us) write_op.cc:301] APPLY: starting
0808 01:10:59.945781 (+ 2791us) tablet.cc:1365] starting BulkCheckPresence
0808 01:10:59.970895 (+ 25114us) tablet.cc:1368] finished BulkCheckPresence
0808 01:11:00.014612 (+ 43717us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 01:11:00.194752 (+180140us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 01:11:00.195231 (+ 479us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0808 01:11:00.195338 (+ 107us) write_op.cc:312] APPLY: finished
0808 01:11:00.195768 (+ 430us) log.cc:842] Serialized 53 byte log entry
0808 01:11:00.331503 (+135735us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:11:00.331727 (+ 224us) write_op.cc:454] Released schema lock
0808 01:11:00.331872 (+ 145us) write_op.cc:341] FINISH: Updating metrics
0808 01:11:00.332436 (+ 564us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:11:00.332493 (+ 57us) write_op.cc:454] Released schema lock
0808 01:11:01.285470 (+952977us) spinlock_profiling.cc:243] Waited 28.4 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
Metrics: {"mutex_wait_us":28912,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":633168,"apply.run_cpu_time_us":20535,"apply.run_wall_time_us":400251,"num_ops":6,"prepare.queue_time_us":791694,"prepare.run_cpu_time_us":13207,"prepare.run_wall_time_us":687026,"raft.queue_time_us":193926,"raft.run_cpu_time_us":8125,"raft.run_wall_time_us":335220,"replication_time_us":1155997,"spinlock_wait_cycles":65397504,"thread_start_us":1364715,"threads_started":5,"wal-append.queue_time_us":474916,"wal-append.run_cpu_time_us":41490,"wal-append.run_wall_time_us":3579736}]]}
I20260808 01:11:03.255211 4233 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:03.255023 2209 raft_consensus.cc:1226] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:03.256186 2209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 745) took 1455 ms. Trace:
I20260808 01:11:03.256508 2209 rpcz_store.cc:276] 0808 01:11:01.801054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.254541 (+1453487us) service_pool.cc:227] Handling call
0808 01:11:03.254859 (+ 318us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:03.255900 (+ 1041us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.256123 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.236654 1340 catalog_manager.cc:5495] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 1 to 2, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 2 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:11:03.258313 1445 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:03.239719 2171 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:03.256506 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:03.261803 1519 raft_consensus.cc:1226] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260808 01:11:03.204563 4165 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:03.264242 1568 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:03.267073 2202 raft_consensus.cc:1261] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:11:03.268575 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:03.271193 1444 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:03.263511 4131 raft_consensus.cc:2781] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:03.273324 1448 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:03.254639 2203 rpcz_store.cc:276] 0808 01:11:01.681884 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.252964 (+1571080us) service_pool.cc:227] Handling call
0808 01:11:03.253294 (+ 330us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:03.254016 (+ 722us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.254217 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.274695 1604 raft_consensus.cc:1226] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:03.277509 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:03.272891 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:03.279860 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:03.281730 2202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 744) took 1642 ms. Trace:
I20260808 01:11:03.282226 2202 rpcz_store.cc:276] 0808 01:11:01.628542 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.251469 (+1622927us) service_pool.cc:227] Handling call
0808 01:11:03.256585 (+ 5116us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:03.270515 (+ 13930us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.270768 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.284476 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:03.285147 1520 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:03.286172 1568 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:03.286716 1443 raft_consensus.cc:2379] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 5: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:03.288652 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:03.293273 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 738) took 3403 ms. Trace:
I20260808 01:11:03.293816 2204 rpcz_store.cc:276] 0808 01:10:59.889854 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.189670 (+3299816us) service_pool.cc:227] Handling call
0808 01:11:03.224549 (+ 34879us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:03.292819 (+ 68270us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:03.292857 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:03.292919 (+ 62us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:03.292979 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.293060 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:03.293204 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.295919 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:03.298769 2207 raft_consensus.cc:1226] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260808 01:11:03.301079 2209 raft_consensus.cc:1226] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:03.276854 3689 raft_consensus.cc:2781] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:03.305419 3689 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:03.305821 3689 raft_consensus.cc:3037] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:03.312587 4192 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:03.313189 4192 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:03.315521 4192 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:03.198215 4099 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:03.273332 4131 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:03.303803 2207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 747) took 1498 ms. Trace:
I20260808 01:11:03.332366 2208 raft_consensus.cc:1261] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:11:03.332274 2207 rpcz_store.cc:276] 0808 01:11:01.801818 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.256570 (+1454752us) service_pool.cc:227] Handling call
0808 01:11:03.298398 (+ 41828us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:03.299356 (+ 958us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.299628 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.304713 2209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 578) took 1451 ms. Trace:
I20260808 01:11:03.319392 2200 raft_consensus.cc:1261] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:11:03.334576 2200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1010) took 1765 ms. Trace:
I20260808 01:11:03.335094 2200 rpcz_store.cc:276] 0808 01:11:01.568649 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.248770 (+1680121us) service_pool.cc:227] Handling call
0808 01:11:03.249117 (+ 347us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:03.334269 (+ 85152us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.334507 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.333801 2209 rpcz_store.cc:276] 0808 01:11:01.853772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.257701 (+1403929us) service_pool.cc:227] Handling call
0808 01:11:03.300882 (+ 43181us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:03.304497 (+ 3615us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.304662 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.298764 3911 raft_consensus.cc:2726] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:03.334343 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:03.333125 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1004) took 1766 ms. Trace:
I20260808 01:11:03.337508 2206 raft_consensus.cc:1226] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEARNER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20260808 01:11:03.322096 1603 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:03.321362 2201 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:11:03.301026 4041 raft_consensus.cc:2781] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:03.341526 2201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 587) took 1485 ms. Trace:
I20260808 01:11:03.342085 2201 rpcz_store.cc:276] 0808 01:11:01.855677 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.105658 (+1249981us) service_pool.cc:227] Handling call
0808 01:11:03.341451 (+235793us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.328941 4040 raft_consensus.cc:2781] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:03.348719 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:03.337553 2208 rpcz_store.cc:276] 0808 01:11:01.566421 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.227381 (+1660960us) service_pool.cc:227] Handling call
0808 01:11:03.329469 (+102088us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:03.332868 (+ 3399us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.333071 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.334080 4131 raft_consensus.cc:3037] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:03.359203 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:03.353875 1525 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:03.360607 1518 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:03.361166 2200 raft_consensus.cc:1226] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:03.361568 1523 raft_consensus.cc:1261] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260808 01:11:03.363049 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 123404us
I20260808 01:11:03.381182 2204 raft_consensus.cc:1261] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:11:03.382412 2202 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEARNER]: Advancing to term 6
I20260808 01:11:03.378468 1522 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:03.378374 1567 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:03.379367 4166 consensus_queue.cc:1046] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.008s
I20260808 01:11:03.391005 1438 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:03.383173 1521 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:11:03.392942 4099 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:03.393661 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:03.394465 1521 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:11:03.362097 4041 raft_consensus.cc:686] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:03.403486 4041 consensus_queue.cc:237] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:03.404335 1519 raft_consensus.cc:1261] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:11:03.407287 2201 raft_consensus.cc:1226] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:03.408156 3861 raft_consensus.cc:2781] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:03.408546 3861 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:03.408876 3861 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:03.420707 2171 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:03.423094 4192 consensus_queue.cc:1035] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:11:03.423540 1490 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:03.424464 4102 consensus_queue.cc:1035] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:11:03.424644 4192 consensus_queue.cc:1035] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:11:03.425853 4102 raft_consensus.cc:2781] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:03.426223 4102 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:03.426422 2204 raft_consensus.cc:1226] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:03.429246 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 743) took 1893 ms. Trace:
I20260808 01:11:03.429843 1604 raft_consensus.cc:1226] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:03.429795 2206 rpcz_store.cc:276] 0808 01:11:01.536233 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.225408 (+1689175us) service_pool.cc:227] Handling call
0808 01:11:03.336252 (+110844us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:03.338063 (+ 1811us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.428946 (+ 90883us) spinlock_profiling.cc:243] Waited 52 ms on lock 0x7b5400502150. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:03.429132 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":119695616}
I20260808 01:11:03.431453 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:03.426649 4102 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:03.446729 1520 raft_consensus.cc:2454] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:03.446738 1444 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:03.451143 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:03.450650 1518 raft_consensus.cc:2454] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:11:03.461570 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:03.474896 1568 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:03.460489 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 68216.8us
I20260808 01:11:03.480135 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:03.481022 1604 raft_consensus.cc:2379] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 5: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:03.481967 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:03.483299 2202 raft_consensus.cc:1261] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEARNER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:11:03.482604 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:03.480931 1596 raft_consensus.cc:1226] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260808 01:11:03.492143 1519 raft_consensus.cc:1261] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:11:03.493361 1523 raft_consensus.cc:2379] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 5: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:03.484714 2202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 579) took 1630 ms. Trace:
I20260808 01:11:03.493748 1517 raft_consensus.cc:2379] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 4: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:03.493888 2202 rpcz_store.cc:276] 0808 01:11:01.854069 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.326583 (+1472514us) service_pool.cc:227] Handling call
0808 01:11:03.349727 (+ 23144us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:03.483719 (+133992us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.484658 (+ 939us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:03.447463 4040 raft_consensus.cc:686] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:03.403084 3772 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:03.500187 1565 leader_election.cc:334] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.429s (SENT)
I20260808 01:11:03.503752 1565 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:03.501911 1594 raft_consensus.cc:1226] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260808 01:11:03.509598 1519 raft_consensus.cc:1226] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:03.473114 3992 raft_consensus.cc:2726] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:03.513715 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:03.484128 4009 raft_consensus.cc:3032] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:11:03.479770 4115 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:03.533576 1489 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:03.535650 1594 raft_consensus.cc:1261] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260808 01:11:03.536823 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 116342us
I20260808 01:11:03.537454 1517 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260808 01:11:03.503417 4140 raft_consensus.cc:2781] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:03.460888 3801 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:03.553609 1567 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:03.495143 4067 consensus_queue.cc:1035] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
W20260808 01:11:03.561554 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 61405.9us
I20260808 01:11:03.496796 4166 consensus_queue.cc:1046] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260808 01:11:03.540009 4198 raft_consensus.cc:2726] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:03.569361 1567 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:03.572999 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:03.526106 4009 raft_consensus.cc:729] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:03.568387 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:03.548010 4234 raft_consensus.cc:2781] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 01:11:03.589042 1490 leader_election.cc:334] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.671s (SENT)
I20260808 01:11:03.597016 1601 raft_consensus.cc:2454] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:03.598331 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:03.599056 2171 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:03.611946 1490 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:03.524480 3907 consensus_queue.cc:1035] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.095s
I20260808 01:11:03.588325 4140 raft_consensus.cc:686] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:03.550043 4235 consensus_queue.cc:1035] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260808 01:11:03.638239 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:03.554368 4188 consensus_queue.cc:1046] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260808 01:11:03.596676 4144 raft_consensus.cc:2781] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:03.601104 4234 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:03.556392 4115 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:03.614245 4057 raft_consensus.cc:2781] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:03.643323 1594 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:11:03.644740 1604 raft_consensus.cc:1226] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:03.651060 1447 raft_consensus.cc:1226] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:03.628449 4170 raft_consensus.cc:2726] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:03.658123 1489 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:03.591150 4185 consensus_queue.cc:1046] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.051s
I20260808 01:11:03.645545 4166 raft_consensus.cc:3032] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Stepping down as leader of term 3
W20260808 01:11:03.664042 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 99319.9us
I20260808 01:11:03.648150 4036 raft_consensus.cc:2726] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:11:03.661124 1565 leader_election.cc:334] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.630s (SENT)
I20260808 01:11:03.687529 1565 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:03.688427 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 97933.6us
W20260808 01:11:03.689836 1487 leader_election.cc:334] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.653s (SENT)
I20260808 01:11:03.653111 4179 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:03.663683 3929 raft_consensus.cc:2781] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:03.652137 4234 raft_consensus.cc:3037] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:11:03.701823 1490 leader_election.cc:334] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.813s (SENT)
I20260808 01:11:03.684266 4166 raft_consensus.cc:729] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:03.686625 4057 raft_consensus.cc:686] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:03.698376 3915 consensus_queue.cc:1046] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.117s
I20260808 01:11:03.705524 1604 raft_consensus.cc:1261] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:11:03.701303 3929 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:03.633229 4040 consensus_queue.cc:237] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:03.711014 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 118013us
I20260808 01:11:03.653439 4146 consensus_queue.cc:1046] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.225s
I20260808 01:11:03.698877 4192 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:03.697757 4179 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:03.719738 3929 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:11:03.757138 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 55432.6us
I20260808 01:11:03.745728 3772 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:03.708617 4199 consensus_queue.cc:1046] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260808 01:11:03.749105 4144 raft_consensus.cc:686] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:03.714926 4098 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:03.695493 4009 consensus_queue.cc:260] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:03.749675 3801 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:03.787307 1490 leader_election.cc:334] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.839s (SENT)
W20260808 01:11:03.790423 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 98686.7us
I20260808 01:11:03.711280 4140 consensus_queue.cc:237] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:03.804955 4239 log.cc:924] Time spent T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20260808 01:11:03.747902 4192 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:03.808203 1596 raft_consensus.cc:1226] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:03.818861 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:03.832413 1447 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
W20260808 01:11:03.810357 4238 log.cc:924] Time spent T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.211s user 0.001s sys 0.001s
I20260808 01:11:03.765139 3689 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:03.836939 2171 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:03.807374 4252 raft_consensus.cc:2726] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:11:03.853004 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 71875us
W20260808 01:11:03.858131 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 197029us
I20260808 01:11:03.860199 2205 raft_consensus.cc:1226] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:03.836139 4036 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:03.840727 4115 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:03.809897 4166 consensus_queue.cc:260] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:03.875633 1568 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:03.787092 4057 consensus_queue.cc:237] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:03.884370 1490 leader_election.cc:334] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.981s (SENT)
I20260808 01:11:03.885069 1490 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:03.867890 4009 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:03.869195 4170 raft_consensus.cc:2781] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:11:03.841120 4131 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:03.892546 4235 raft_consensus.cc:2726] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:11:03.886792 4244 log.cc:924] Time spent T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.203s user 0.000s sys 0.001s
W20260808 01:11:03.897635 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 60791.1us
I20260808 01:11:03.905536 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:03.906354 1523 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:11:03.907038 4166 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:03.912976 3689 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:11:03.915568 4253 log.cc:924] Time spent T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260808 01:11:03.919610 1599 raft_consensus.cc:1203] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[2.2-2.2] Dedup: 2.2->[]
I20260808 01:11:03.920344 1599 raft_consensus.cc:1226] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 01:11:03.920912 4170 raft_consensus.cc:686] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
W20260808 01:11:03.921031 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 395005us
I20260808 01:11:03.922037 4170 consensus_queue.cc:237] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:03.922770 4263 raft_consensus.cc:2781] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:03.923209 4263 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:03.923552 4263 raft_consensus.cc:3037] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:03.924880 4098 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:03.926155 4192 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:03.926785 3861 consensus_meta.cc:287] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.517s user 0.012s sys 0.000s
I20260808 01:11:03.927239 3861 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:03.929359 3861 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:03.929509 2205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 741) took 2876 ms. Trace:
I20260808 01:11:03.929891 2205 rpcz_store.cc:276] 0808 01:11:01.053440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.215520 (+2162080us) service_pool.cc:227] Handling call
0808 01:11:03.272199 (+ 56679us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:03.860755 (+588556us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.929256 (+ 68501us) spinlock_profiling.cc:243] Waited 588 ms on lock 0x7b5400446454. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:03.929451 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1351720320}
I20260808 01:11:03.931376 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:03.932071 1438 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:11:03.934465 1490 leader_election.cc:334] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.653s (SENT)
I20260808 01:11:03.934916 1490 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:03.939790 2201 raft_consensus.cc:1226] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260808 01:11:03.943123 4197 raft_consensus.cc:3032] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:03.943553 4197 raft_consensus.cc:729] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:03.944154 4179 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:03.944271 3801 consensus_queue.cc:1035] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260808 01:11:03.954614 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 51828.6us
I20260808 01:11:03.955677 2171 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:03.932803 4246 log.cc:924] Time spent T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.225s user 0.000s sys 0.001s
I20260808 01:11:03.898476 4115 raft_consensus.cc:3032] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:03.966845 1448 raft_consensus.cc:1226] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:03.966585 1524 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:03.969915 1444 raft_consensus.cc:1261] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:11:03.888651 4036 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:03.868598 4144 consensus_queue.cc:237] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:03.965926 3801 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:03.965499 3992 raft_consensus.cc:2726] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:03.990319 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:03.965698 4115 raft_consensus.cc:729] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:04.006109 3689 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:04.006987 3689 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:04.036685 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:11:03.993258 4102 consensus_meta.cc:287] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.506s user 0.009s sys 0.001s
I20260808 01:11:04.041366 1448 raft_consensus.cc:1226] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:03.982324 4131 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:04.051219 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:04.053334 1448 raft_consensus.cc:2354] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4.
I20260808 01:11:03.971189 3772 consensus_queue.cc:1046] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
I20260808 01:11:04.056085 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:04.057204 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 67001us
I20260808 01:11:04.059490 3801 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:04.108090 2172 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:04.121399 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:04.056198 4188 raft_consensus.cc:2726] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:04.123054 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:04.133241 2202 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:04.134166 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:04.134948 1444 raft_consensus.cc:2379] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
W20260808 01:11:04.140697 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 206244us
I20260808 01:11:04.139432 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:04.143204 2205 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260808 01:11:04.143786 1448 raft_consensus.cc:2354] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7.
I20260808 01:11:04.142928 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:04.148676 1517 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Advancing to term 10
W20260808 01:11:04.147315 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 110402us
I20260808 01:11:03.990921 4267 consensus_queue.cc:1035] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.143s
I20260808 01:11:04.107282 4185 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:04.122360 3801 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:04.122848 3992 raft_consensus.cc:2781] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:04.133342 3772 raft_consensus.cc:3032] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:04.196334 2202 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:04.155646 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:04.150594 4184 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:04.207639 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:04.208176 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:04.208787 1521 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:04.211393 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:04.228163 1443 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:04.229175 1444 raft_consensus.cc:1261] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:11:04.206202 3772 raft_consensus.cc:729] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:04.206580 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
W20260808 01:11:04.215698 4165 consensus_meta.cc:287] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.929s user 0.008s sys 0.006s
W20260808 01:11:04.261458 1490 leader_election.cc:334] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 2.208s (SENT)
I20260808 01:11:04.250780 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:04.263427 2171 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:04.048642 4102 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:04.145246 4115 consensus_queue.cc:260] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:04.266857 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:04.272256 1448 raft_consensus.cc:1261] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:11:04.277844 1518 raft_consensus.cc:2379] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 4: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:04.230383 3992 raft_consensus.cc:686] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:04.203400 4185 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:04.273125 2203 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:04.266846 4036 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:04.292253 1602 raft_consensus.cc:1226] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260808 01:11:04.294533 1438 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 10.
W20260808 01:11:04.296209 4272 log.cc:924] Time spent T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.090s user 0.001s sys 0.000s
W20260808 01:11:04.297627 4271 log.cc:924] Time spent T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20260808 01:11:04.297518 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:04.297503 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:04.276254 4131 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:04.299315 4131 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:04.301254 4102 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:04.301862 4263 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:04.304160 4263 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:11:04.304778 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 92887.1us
I20260808 01:11:04.306308 1443 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:04.324155 2171 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:04.324374 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:04.327397 4275 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:04.307910 1517 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 10.
W20260808 01:11:04.309427 1524 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58280 (request call id 1873) took 3232 ms (client timeout 1913 ms). Trace:
I20260808 01:11:04.330318 4275 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:04.311719 1516 raft_consensus.cc:1261] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260808 01:11:04.331390 4247 log.cc:924] Time spent T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.593s user 0.006s sys 0.006s
I20260808 01:11:04.281853 3689 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:04.332777 3689 raft_consensus.cc:2726] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:04.333379 1516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1908) took 1443 ms. Trace:
I20260808 01:11:04.334017 4275 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:04.331070 1524 rpcz_store.cc:269] 0808 01:11:01.077107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:01.078061 (+ 954us) service_pool.cc:227] Handling call
0808 01:11:02.709447 (+1631386us) spinlock_profiling.cc:243] Waited 33.4 ms on lock 0x7b5800035d98. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adf851 00007fe349adf702 00007fe3529cfb09 00007fe35190b730 00007fe3519d179b 00007fe3519d1821 00007fe3519811b0 00007fe351980a94 00007fe34b068c04 00007fe34b068b92
0808 01:11:04.309219 (+1599772us) spinlock_profiling.cc:243] Waited 915 ms on lock 0x7b54001a04d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:04.309336 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2181360256}
I20260808 01:11:04.333739 1516 rpcz_store.cc:276] 0808 01:11:02.890305 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.097280 (+206975us) service_pool.cc:227] Handling call
0808 01:11:04.311303 (+1214023us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:04.331566 (+ 20263us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:04.333180 (+ 1614us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54001a04d0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:04.333322 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2791106944}
I20260808 01:11:04.312686 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:04.313922 4197 consensus_queue.cc:260] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:04.317418 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:04.318089 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:04.318104 1439 raft_consensus.cc:1261] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:11:04.319442 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:04.285831 4234 consensus_meta.cc:287] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.552s user 0.012s sys 0.000s
I20260808 01:11:04.299686 4131 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:04.322855 4113 raft_consensus.cc:3032] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Stepping down as leader of term 6
W20260808 01:11:04.323105 3929 consensus_meta.cc:287] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.559s user 0.005s sys 0.005s
I20260808 01:11:04.349063 3929 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:04.349947 4260 log.cc:924] Time spent T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.424s user 0.000s sys 0.001s
I20260808 01:11:04.351219 3929 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:04.341173 4113 raft_consensus.cc:729] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:04.353106 4113 consensus_queue.cc:260] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:04.354089 4113 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:04.357029 4197 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:04.358755 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:04.359455 1447 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:04.361249 3992 consensus_queue.cc:237] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:04.332816 1521 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
W20260808 01:11:04.314276 4240 log.cc:924] Time spent T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
W20260808 01:11:04.339291 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 77898.6us
I20260808 01:11:04.369222 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1026) took 1320 ms. Trace:
I20260808 01:11:04.369661 2208 rpcz_store.cc:276] 0808 01:11:03.048651 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.355436 (+306785us) service_pool.cc:227] Handling call
0808 01:11:03.378572 (+ 23136us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:03.520319 (+141747us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:03.520351 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:03.538991 (+ 18640us) log.cc:842] Serialized 26 byte log entry
0808 01:11:03.654741 (+115750us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:03.805723 (+150982us) log.cc:842] Serialized 14 byte log entry
0808 01:11:03.806334 (+ 611us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.806409 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:04.354358 (+547949us) spinlock_profiling.cc:243] Waited 2.54 ms on lock 0x7b1800312580. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349b17638 00007fe349b1ab11 00007fe349b1aac9 00007fe349b1aa59 00007fe349b1aa21 00007fe349b19d1d 00007fe352d7b204 00007fe352d7b039 00007fe349af9dc6
0808 01:11:04.363622 (+ 9264us) raft_consensus.cc:1671] finished
0808 01:11:04.369023 (+ 5401us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:04.369162 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":44539,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":833,"raft.queue_time_us":39262,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":11134,"spinlock_wait_cycles":5832064,"thread_start_us":187873,"threads_started":2,"wal-append.queue_time_us":184503}
I20260808 01:11:04.370203 1410 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:04.245919 4249 consensus_queue.cc:1046] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.220s
I20260808 01:11:04.343272 4127 raft_consensus.cc:3032] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:04.373147 4127 raft_consensus.cc:729] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:04.374082 4127 consensus_queue.cc:260] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:04.344578 1446 raft_consensus.cc:1226] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 01:11:04.375253 4127 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:11:04.347141 4241 log.cc:924] Time spent T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.613s user 0.000s sys 0.001s
W20260808 01:11:04.375813 4270 log.cc:924] Time spent T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20260808 01:11:04.229723 4251 consensus_queue.cc:1046] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.342s
I20260808 01:11:04.344657 2201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 761) took 1070 ms. Trace:
I20260808 01:11:04.340639 4234 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:04.372236 1440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2033) took 1321 ms. Trace:
I20260808 01:11:04.377113 2201 rpcz_store.cc:276] 0808 01:11:03.273761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.409179 (+135418us) service_pool.cc:227] Handling call
0808 01:11:03.409476 (+ 297us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:03.953593 (+544117us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:04.344406 (+390813us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x7b54007c1d54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:04.344594 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1203325312}
I20260808 01:11:04.214708 4184 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:11:04.407148 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 193531us
I20260808 01:11:04.334051 4185 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:04.411755 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:11:04.405102 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 354773us
I20260808 01:11:04.411665 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:04.385422 1440 rpcz_store.cc:276] 0808 01:11:03.051243 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.056369 (+ 5126us) service_pool.cc:227] Handling call
0808 01:11:03.176473 (+120104us) spinlock_profiling.cc:243] Waited 2.62 ms on lock 0x7b5800003698. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adf851 00007fe349adf702 00007fe3529cfb09 00007fe35190b730 00007fe3519d179b 00007fe3519d1821 00007fe3519952b0 00007fe3519807f9 00007fe3519802a2 00007fe34b066684
0808 01:11:03.176664 (+ 191us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:03.198343 (+ 21679us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:03.198374 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 01:11:03.224037 (+ 25663us) log.cc:842] Serialized 52 byte log entry
0808 01:11:03.372480 (+148443us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:03.442589 (+ 70109us) log.cc:842] Serialized 14 byte log entry
0808 01:11:03.477701 (+ 35112us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.477779 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:04.152383 (+674604us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b1800317800. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349b17638 00007fe349b1ab11 00007fe349b1aac9 00007fe349b1aa59 00007fe349b1aa21 00007fe349b19d1d 00007fe352d7b204 00007fe352d7b039 00007fe349af9dc6
0808 01:11:04.372033 (+219650us) raft_consensus.cc:1671] finished
0808 01:11:04.372057 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:04.372184 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34958,"prepare.queue_time_us":69920,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":5539,"raft.queue_time_us":40209,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":17177,"spinlock_wait_cycles":33405184,"thread_start_us":140229,"threads_started":1,"wal-append.queue_time_us":265672}
I20260808 01:11:04.424947 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:04.429538 4234 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:04.379700 4146 raft_consensus.cc:2741] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260808 01:11:04.425204 1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2500) took 1299 ms. Trace:
W20260808 01:11:04.442899 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 51787.2us
I20260808 01:11:04.445225 1490 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:04.430913 4251 raft_consensus.cc:3032] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:04.458323 1408 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:04.459784 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:04.460423 1446 raft_consensus.cc:3037] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:04.440021 1597 rpcz_store.cc:276] 0808 01:11:03.113221 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.113383 (+ 162us) service_pool.cc:227] Handling call
0808 01:11:03.135873 (+ 22490us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:03.164656 (+ 28783us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:03.164688 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 01:11:03.187938 (+ 23250us) log.cc:842] Serialized 52 byte log entry
0808 01:11:03.495944 (+308006us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:03.495986 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:03.496054 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:04.346363 (+850309us) spinlock_profiling.cc:243] Waited 4.61 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:04.376722 (+ 30359us) raft_consensus.cc:1671] finished
0808 01:11:04.376742 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:04.412236 (+ 35494us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10353,"raft.queue_time_us":4875,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":11881,"spinlock_wait_cycles":10600320,"thread_start_us":221787,"threads_started":1,"wal-append.queue_time_us":465111}
W20260808 01:11:04.463428 1565 leader_election.cc:334] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.504s (SENT)
I20260808 01:11:04.464082 1565 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:04.413166 4041 consensus_queue.cc:1046] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260808 01:11:04.341764 3772 consensus_queue.cc:260] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:04.457005 4234 raft_consensus.cc:2726] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:04.448927 4251 raft_consensus.cc:729] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:04.479102 2201 raft_consensus.cc:1226] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:04.491201 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:04.492384 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:04.496399 2200 raft_consensus.cc:1226] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:04.474642 4087 raft_consensus.cc:3032] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:11:04.479957 4134 raft_consensus.cc:2781] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election won for term 5
W20260808 01:11:04.505849 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 73110.6us
I20260808 01:11:04.467927 3801 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20260808 01:11:04.514282 1601 raft_consensus.cc:1226] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:04.473501 4249 consensus_queue.cc:1035] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.026s
I20260808 01:11:04.520507 1567 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:04.520725 4131 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:04.521445 4134 raft_consensus.cc:686] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:04.494062 4263 raft_consensus.cc:2726] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:04.518236 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:04.484169 3915 consensus_queue.cc:1046] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260808 01:11:04.534585 1596 raft_consensus.cc:2354] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 9. Current term is 10.
I20260808 01:11:04.535919 1567 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:04.504176 4087 raft_consensus.cc:729] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:04.533547 4131 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:04.539026 1490 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:04.534201 4282 log.cc:924] Time spent T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
I20260808 01:11:04.546736 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:04.554962 1601 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:04.543552 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:04.559209 1598 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
W20260808 01:11:04.552245 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 74452us
W20260808 01:11:04.541275 4057 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.009s sys 0.024s
I20260808 01:11:04.530269 4146 consensus_queue.cc:1035] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260808 01:11:04.553681 4259 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:11:04.553429 4256 raft_consensus.cc:2781] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:04.583026 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:11:04.567921 4057 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s user 0.009s sys 0.025s
I20260808 01:11:04.564028 4275 raft_consensus.cc:2726] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:04.553157 4199 consensus_queue.cc:1035] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260808 01:11:04.617508 1601 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
W20260808 01:11:04.622500 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 298395us
I20260808 01:11:04.596477 4256 raft_consensus.cc:686] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:04.596698 4184 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:04.604817 4259 raft_consensus.cc:686] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 10 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
W20260808 01:11:04.632187 1487 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.000s sys 0.016s
W20260808 01:11:04.655011 1487 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.539s user 0.000s sys 0.017s
W20260808 01:11:04.655429 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 539490us
W20260808 01:11:04.707139 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 205159us
W20260808 01:11:04.739154 4286 log.cc:924] Time spent T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
I20260808 01:11:04.740415 4259 consensus_queue.cc:237] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:04.757416 4274 raft_consensus.cc:3032] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:04.757997 4274 raft_consensus.cc:729] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
W20260808 01:11:04.760584 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 247316us
W20260808 01:11:04.761934 4288 log.cc:924] Time spent T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
I20260808 01:11:04.762032 1446 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
W20260808 01:11:04.769675 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 197228us
W20260808 01:11:04.778575 1487 leader_election.cc:334] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.714s (SENT)
I20260808 01:11:04.779163 1487 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:04.561616 4251 consensus_queue.cc:260] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:04.777667 2171 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:04.782681 2206 raft_consensus.cc:1226] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:04.604930 4134 consensus_queue.cc:237] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:04.759008 4274 consensus_queue.cc:260] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:04.775607 4263 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:04.607743 4087 consensus_queue.cc:260] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:04.788580 4131 raft_consensus.cc:2781] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election won for term 7
W20260808 01:11:04.797529 1565 leader_election.cc:334] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.036s (SENT)
I20260808 01:11:04.805634 1565 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:04.809803 2171 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:04.814170 4292 log.cc:924] Time spent T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260808 01:11:04.814615 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:04.815304 1516 raft_consensus.cc:3037] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:04.816617 1523 raft_consensus.cc:1261] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260808 01:11:04.817020 1523 pending_rounds.cc:76] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5: Aborting all ops after (but not including) 4
I20260808 01:11:04.817267 1523 pending_rounds.cc:97] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260808 01:11:04.817535 1523 raft_consensus.cc:2866] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:04.819589 1520 raft_consensus.cc:1226] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: []
W20260808 01:11:04.824517 4285 log.cc:924] Time spent T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.304s user 0.000s sys 0.003s
W20260808 01:11:04.824792 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 54112.7us
I20260808 01:11:04.825608 4127 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20260808 01:11:04.825613 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 250959us
I20260808 01:11:04.855929 1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2520) took 1061 ms. Trace:
I20260808 01:11:04.861598 1410 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:04.792261 4275 raft_consensus.cc:2726] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:04.869095 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:04.869833 1522 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:04.870213 3801 consensus_queue.cc:1035] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:11:04.799233 4256 consensus_queue.cc:237] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:04.878350 2203 consensus_meta.cc:287] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.526s user 0.005s sys 0.005s
I20260808 01:11:04.878798 2203 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
W20260808 01:11:04.891368 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 60415us
I20260808 01:11:04.890189 2171 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:04.856588 1604 rpcz_store.cc:276] 0808 01:11:03.794474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:03.804341 (+ 9867us) service_pool.cc:227] Handling call
0808 01:11:03.804762 (+ 421us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:03.879465 (+ 74703us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:03.879488 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:04.476276 (+596788us) spinlock_profiling.cc:243] Waited 1.46 ms on lock 0x7b2c00042c88. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34b422e11 00007fe34b432442 00007fe34ca7ac11 00007fe34ca3f58a 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772
0808 01:11:04.476797 (+ 521us) log.cc:842] Serialized 26 byte log entry
0808 01:11:04.595044 (+118247us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:04.595075 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:04.595140 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:04.824400 (+229260us) raft_consensus.cc:1671] finished
0808 01:11:04.824417 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:04.855783 (+ 31366us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4389,"raft.queue_time_us":85111,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":2059,"spinlock_wait_cycles":3353984,"thread_start_us":187772,"threads_started":2,"wal-append.queue_time_us":326635}
I20260808 01:11:04.830852 4087 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:04.892551 1522 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:04.919667 1516 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
W20260808 01:11:04.883422 4197 consensus_meta.cc:287] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.526s user 0.008s sys 0.003s
I20260808 01:11:04.921597 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:04.923784 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:04.863700 4274 raft_consensus.cc:3037] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:04.799752 4263 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:04.924448 1604 raft_consensus.cc:3037] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:04.924492 1520 raft_consensus.cc:1261] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260808 01:11:04.925961 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 128462us
W20260808 01:11:04.928236 1565 leader_election.cc:334] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.167s (SENT)
I20260808 01:11:04.928649 1565 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:04.926064 1519 raft_consensus.cc:1261] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:11:04.862639 4131 raft_consensus.cc:686] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:04.908159 4199 raft_consensus.cc:2726] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:04.925400 1567 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:04.908495 3689 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:04.932060 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 71220.4us
I20260808 01:11:04.933377 2171 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:04.933630 4289 raft_consensus.cc:2726] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:04.934365 4289 raft_consensus.cc:2726] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:11:04.934907 3689 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:04.829042 3915 consensus_queue.cc:1159] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" responder_term: 3 status { last_received { term: 2 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:11:04.937325 4115 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:04.937525 3689 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:04.941268 1568 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:04.944414 4041 raft_consensus.cc:2781] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:04.944844 4041 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:04.924687 4170 raft_consensus.cc:2726] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:04.888396 4295 consensus_queue.cc:1046] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.057s
I20260808 01:11:04.945654 4170 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:04.946218 4170 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:04.947455 4263 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:04.947711 1440 raft_consensus.cc:1261] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:11:04.952349 1447 raft_consensus.cc:1261] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 01:11:04.943998 4269 consensus_queue.cc:1035] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260808 01:11:04.945298 4041 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:04.971174 1489 leader_election.cc:398] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 pre-election: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 9. Current term is 10.
I20260808 01:11:04.971176 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:04.971899 1489 leader_election.cc:401] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:11:04.972581 1447 raft_consensus.cc:2396] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
W20260808 01:11:04.978335 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 52926.5us
I20260808 01:11:04.982304 1440 raft_consensus.cc:1261] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260808 01:11:04.985383 1490 leader_election.cc:334] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 2.199s (SENT)
I20260808 01:11:04.985838 1490 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:04.978384 4234 raft_consensus.cc:2726] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 9. Current term is 10.
I20260808 01:11:04.990465 1444 raft_consensus.cc:3032] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:04.991024 1444 raft_consensus.cc:729] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:04.992103 1444 consensus_queue.cc:260] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:04.993407 1444 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:11:05.006980 1487 leader_election.cc:334] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.128s (SENT)
I20260808 01:11:05.007645 1487 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:05.008586 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 80367.8us
I20260808 01:11:05.032332 1604 raft_consensus.cc:2454] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:11:05.031502 1444 raft_consensus.cc:1261] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 2 index: 2. (term mismatch)
I20260808 01:11:05.009240 4255 raft_consensus.cc:2781] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:05.038691 4255 raft_consensus.cc:686] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:05.032603 4170 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:05.039738 4255 consensus_queue.cc:237] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:05.037514 4192 raft_consensus.cc:2726] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:11:05.040897 4099 raft_consensus.cc:2726] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260808 01:11:05.043023 1565 leader_election.cc:334] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.920s (SENT)
I20260808 01:11:05.043514 1565 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:05.054771 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 121471us
I20260808 01:11:05.035524 1444 pending_rounds.cc:76] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d: Aborting all ops after (but not including) 1
I20260808 01:11:05.057994 2171 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:05.059444 2171 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:05.060479 3689 raft_consensus.cc:2726] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:11:05.061338 3689 raft_consensus.cc:2781] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:11:05.061690 3689 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:05.062114 3689 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:11:05.035785 4263 raft_consensus.cc:2726] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:11:05.060375 4262 consensus_queue.cc:1035] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:11:05.064379 4263 raft_consensus.cc:2726] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:04.949747 4131 consensus_queue.cc:237] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:05.065626 1444 pending_rounds.cc:97] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260808 01:11:05.066444 2171 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
W20260808 01:11:05.044394 4270 log.cc:924] Time spent T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260808 01:11:05.069948 1444 raft_consensus.cc:2866] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:11:05.083936 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 107344us
I20260808 01:11:05.016856 4057 consensus_queue.cc:1035] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.054s
I20260808 01:11:05.032228 3795 consensus_queue.cc:1035] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
W20260808 01:11:05.094758 1565 leader_election.cc:334] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.600s (SENT)
I20260808 01:11:05.101025 1565 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:05.096187 4289 raft_consensus.cc:2781] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:05.108585 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:05.109442 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.110250 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.110952 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:05.111641 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.134866 2209 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:05.134960 2200 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:05.112389 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.113071 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.113795 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:05.116583 4284 log.cc:924] Time spent T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.601s user 0.006s sys 0.009s
I20260808 01:11:05.136668 2207 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:05.119702 4165 raft_consensus.cc:3032] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Stepping down as leader of term 3
W20260808 01:11:05.121387 4290 log.cc:924] Time spent T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.358s user 0.000s sys 0.011s
I20260808 01:11:05.125761 4041 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:05.137106 2202 raft_consensus.cc:2379] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 7: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:05.136636 2204 raft_consensus.cc:2379] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
W20260808 01:11:05.147166 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 161804us
I20260808 01:11:05.118434 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.144479 2201 raft_consensus.cc:2396] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:11:05.143875 1489 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:05.150355 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.148559 1594 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:05.148568 2208 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
W20260808 01:11:05.149456 1490 leader_election.cc:334] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 2.238s (SENT)
I20260808 01:11:05.151973 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.152680 1489 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:05.152722 2207 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:11:05.154699 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.155411 2200 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:05.144773 4165 raft_consensus.cc:729] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:05.157395 4234 raft_consensus.cc:2781] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:05.160264 4234 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:05.160656 4234 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:05.160347 4165 consensus_queue.cc:260] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:05.161713 4165 raft_consensus.cc:3037] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:05.162451 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:05.162616 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:05.163328 2208 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:05.164319 2206 raft_consensus.cc:2379] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:05.164829 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:05.144805 3861 raft_consensus.cc:2726] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:05.164397 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.167780 2204 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:11:05.165444 2201 raft_consensus.cc:2362] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3. Re-sending same reply.
I20260808 01:11:05.145920 4289 raft_consensus.cc:686] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:05.169348 1521 raft_consensus.cc:1261] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:11:05.171567 2202 raft_consensus.cc:1261] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:11:05.144888 2203 raft_consensus.cc:2362] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4. Re-sending same reply.
I20260808 01:11:05.179046 2207 raft_consensus.cc:1261] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260808 01:11:05.176884 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 94029.6us
W20260808 01:11:05.180591 1565 leader_election.cc:334] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.924s (SENT)
I20260808 01:11:05.180974 1565 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:05.176137 2201 pending_rounds.cc:76] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd: Aborting all ops after (but not including) 5
I20260808 01:11:05.189811 2201 pending_rounds.cc:97] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260808 01:11:05.191581 2201 raft_consensus.cc:2866] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:05.192690 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.194761 4263 raft_consensus.cc:3037] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:11:05.208269 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 346720us
I20260808 01:11:05.208536 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.209031 4249 raft_consensus.cc:2726] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:11:05.209357 2206 raft_consensus.cc:2379] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:05.211421 3801 raft_consensus.cc:3032] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:11:05.211861 3801 raft_consensus.cc:729] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:05.195784 4041 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:05.200307 4289 consensus_queue.cc:237] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:05.212829 3801 consensus_queue.cc:260] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:05.214278 3801 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:05.214771 2209 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:05.162274 4269 consensus_queue.cc:1035] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260808 01:11:05.218155 2202 raft_consensus.cc:2362] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4. Re-sending same reply.
W20260808 01:11:05.218853 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 325601us
I20260808 01:11:05.177609 4252 consensus_queue.cc:1035] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.027s
I20260808 01:11:05.190426 2204 raft_consensus.cc:1261] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:11:05.226227 1444 raft_consensus.cc:1261] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:11:05.189813 2200 raft_consensus.cc:1226] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:05.227188 2208 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
W20260808 01:11:05.192183 3992 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.022s sys 0.037s
I20260808 01:11:05.256445 1408 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:05.257102 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:05.260324 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.260530 1446 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:05.257570 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:05.189682 4192 raft_consensus.cc:2781] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:05.260972 2202 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:05.264494 1489 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:11:05.255949 3992 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.729s user 0.023s sys 0.038s
I20260808 01:11:05.224176 3772 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:11:05.272184 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 91619us
I20260808 01:11:05.236461 4251 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:11:05.290634 1487 leader_election.cc:334] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.432s (SENT)
I20260808 01:11:05.292295 1487 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:05.292742 1410 leader_election.cc:398] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 4. Current term is 5.
I20260808 01:11:05.293123 1410 leader_election.cc:401] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260808 01:11:05.243535 4009 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:05.249140 4295 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:05.288408 4146 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:05.299202 4009 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:05.305498 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.297066 4113 raft_consensus.cc:2781] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:05.306203 2202 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:05.248868 4269 consensus_queue.cc:1035] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260808 01:11:05.299911 3689 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:05.316359 4099 raft_consensus.cc:2726] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:11:05.265791 4098 consensus_queue.cc:1035] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
I20260808 01:11:05.258790 4067 consensus_queue.cc:1035] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20260808 01:11:05.297261 4192 raft_consensus.cc:686] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:05.341260 4192 consensus_queue.cc:237] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:05.343854 3748 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:05.307006 4263 raft_consensus.cc:2726] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:05.344483 3748 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:05.345451 1444 raft_consensus.cc:1261] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260808 01:11:05.347095 3748 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:05.304939 4291 raft_consensus.cc:2781] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:05.307587 3992 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:05.333073 4113 raft_consensus.cc:686] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:05.305321 4146 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:05.339562 4009 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:05.305605 4295 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:05.468971 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 178339us
I20260808 01:11:05.347761 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:05.469376 3689 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:05.470568 1447 raft_consensus.cc:2396] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:11:05.470796 3992 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:05.322448 4249 raft_consensus.cc:2726] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 4. Current term is 5.
I20260808 01:11:05.446514 4234 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:05.477627 2207 raft_consensus.cc:1261] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:11:05.478598 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:05.435252 4328 log.cc:924] Time spent T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20260808 01:11:05.417547 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 153170us
I20260808 01:11:05.480703 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:05.483471 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:05.485468 4295 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:05.486178 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:05.487628 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:05.398479 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 133201us
I20260808 01:11:05.492368 4115 consensus_queue.cc:1035] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:11:05.493708 1410 leader_election.cc:398] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4.
I20260808 01:11:05.493968 4127 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:05.494539 4127 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:05.495152 1410 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:05.396260 4296 log.cc:924] Time spent T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
W20260808 01:11:05.497618 4309 log.cc:924] Time spent T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.349s user 0.000s sys 0.002s
I20260808 01:11:05.477435 4146 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:05.478152 2203 raft_consensus.cc:1261] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:11:05.478332 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:05.502058 2200 raft_consensus.cc:2362] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4. Re-sending same reply.
I20260808 01:11:05.478906 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.502708 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:05.502959 2205 raft_consensus.cc:2362] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5. Re-sending same reply.
I20260808 01:11:05.469461 1446 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:05.503378 1519 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:11:05.479372 4234 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:05.396397 2209 raft_consensus.cc:1226] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 1. Current term is 2. Ops: []
I20260808 01:11:05.499652 1443 raft_consensus.cc:1261] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 01:11:05.507544 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:05.508347 1604 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:11:05.507755 1487 leader_election.cc:334] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.469s (SENT)
I20260808 01:11:05.509438 1487 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:05.510761 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.473284 3992 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:05.524034 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:05.525167 1600 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:11:05.526731 1602 raft_consensus.cc:1226] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:05.527781 1604 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:11:05.533011 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:05.533771 1603 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:11:05.526957 1568 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:05.543148 3861 raft_consensus.cc:2781] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:05.543768 3861 raft_consensus.cc:686] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
W20260808 01:11:05.544771 4324 log.cc:924] Time spent T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.197s user 0.001s sys 0.000s
I20260808 01:11:05.545753 4331 consensus_queue.cc:1035] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
W20260808 01:11:05.546869 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 228611us
I20260808 01:11:05.548614 1521 raft_consensus.cc:1261] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260808 01:11:05.534016 4234 raft_consensus.cc:2726] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:05.544795 3861 consensus_queue.cc:237] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:05.561460 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.513799 4113 consensus_queue.cc:237] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:05.567371 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:05.568251 1522 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:11:05.583770 1489 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:05.586575 1525 raft_consensus.cc:1261] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:11:05.589341 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:05.591393 1603 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:11:05.596390 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 101303us
I20260808 01:11:05.597384 4146 raft_consensus.cc:2726] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:05.597911 1410 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:05.599761 1448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:55036 (request call id 874) took 1032 ms. Trace:
I20260808 01:11:05.600221 1448 rpcz_store.cc:276] 0808 01:11:04.567123 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:04.567375 (+ 252us) service_pool.cc:227] Handling call
0808 01:11:05.599161 (+1031786us) spinlock_profiling.cc:243] Waited 637 ms on lock 0x7b540014d7d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4af0 00007fe34ca47f2a 00007fe351984c6e 00007fe34b07b807 00007fe34b07b792 00007fe34b07b6c1 00007fe34b07b63c 00007fe34b07a8f2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:05.599659 (+ 498us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1465646976}
I20260808 01:11:05.601747 3907 consensus_queue.cc:1035] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:11:05.610378 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:05.611874 1522 raft_consensus.cc:2379] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 4: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:05.610060 1603 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:05.554805 4067 consensus_queue.cc:1035] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260808 01:11:05.643203 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:05.644045 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:05.618745 4333 raft_consensus.cc:2781] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:05.626885 4291 raft_consensus.cc:686] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:05.649187 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:05.605919 4293 consensus_queue.cc:1035] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260808 01:11:05.658524 1522 raft_consensus.cc:1261] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:11:05.653098 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:05.663152 1599 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:05.664681 1594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2220) took 1185 ms. Trace:
I20260808 01:11:05.665126 1594 rpcz_store.cc:276] 0808 01:11:04.471434 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:04.480006 (+ 8572us) service_pool.cc:227] Handling call
0808 01:11:05.657137 (+1177131us) spinlock_profiling.cc:243] Waited 601 ms on lock 0x7b54002ad0d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:05.657256 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1381489408}
I20260808 01:11:05.669970 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:05.626662 4332 consensus_queue.cc:1035] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260808 01:11:05.690110 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:05.650573 4249 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:05.690984 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:05.632076 4199 consensus_queue.cc:1035] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.041s
I20260808 01:11:05.695673 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:05.657671 4131 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:05.696373 1442 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:05.700829 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:05.702080 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:05.645696 4099 consensus_queue.cc:1035] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260808 01:11:05.703663 1443 raft_consensus.cc:2396] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:11:05.680991 3801 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:05.708045 1603 raft_consensus.cc:2379] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 4: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:05.684820 4189 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:05.656149 4269 consensus_queue.cc:1035] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.041s
I20260808 01:11:05.687410 4334 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:05.717008 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:05.717741 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:05.718637 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:05.729948 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:05.731670 1516 raft_consensus.cc:1261] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:11:05.750368 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:05.729635 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:05.758644 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:05.703500 3772 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:05.762358 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:05.766299 4348 log.cc:924] Time spent T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.090s user 0.000s sys 0.004s
I20260808 01:11:05.761070 1439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55042 (request call id 2355) took 1440 ms. Trace:
I20260808 01:11:05.773156 1602 raft_consensus.cc:1226] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 01:11:05.774343 1410 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:05.774477 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:05.775194 1603 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:05.774892 3772 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:05.776194 4341 log.cc:924] Time spent T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
I20260808 01:11:05.776432 4087 raft_consensus.cc:2726] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:05.776342 4319 consensus_queue.cc:1035] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.123s
I20260808 01:11:05.777117 3772 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:05.762593 3992 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:05.777984 3992 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:05.769158 4334 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:05.779426 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:05.780220 4334 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:05.780609 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:05.781194 1613 tablet_copy_service.cc:342] P a4862d0938c84fc4bd840e4c17acae78: Request end of tablet copy session 95b661b03f1d460b9657e61d4b0247cd-c12c84f631ee4ca7b405e28317db1b8a received from {username='slave'} at 127.0.0.1:34182
I20260808 01:11:05.781498 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:05.782436 3689 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:05.782750 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:05.782927 3689 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:05.783352 1446 raft_consensus.cc:2454] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:05.784089 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:05.706248 4131 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:05.781205 1440 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:05.781739 1613 tablet_copy_service.cc:433] P a4862d0938c84fc4bd840e4c17acae78: ending tablet copy session 95b661b03f1d460b9657e61d4b0247cd-c12c84f631ee4ca7b405e28317db1b8a on tablet c12c84f631ee4ca7b405e28317db1b8a with peer 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:05.790705 1599 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:05.770223 1439 rpcz_store.cc:276] 0808 01:11:04.321135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:04.346964 (+ 25829us) service_pool.cc:227] Handling call
0808 01:11:04.347345 (+ 381us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:04.442455 (+ 95110us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:04.442502 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:04.456704 (+ 14202us) log.cc:842] Serialized 26 byte log entry
0808 01:11:04.614472 (+157768us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:05.085648 (+471176us) log.cc:842] Serialized 14 byte log entry
0808 01:11:05.212148 (+126500us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:05.751668 (+539520us) spinlock_profiling.cc:243] Waited 72.8 ms on lock 0x7b540002e4d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:05.751713 (+ 45us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:05.751760 (+ 47us) raft_consensus.cc:1671] finished
0808 01:11:05.760808 (+ 9048us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:05.761000 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":140061,"prepare.queue_time_us":316815,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":140862,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":18560,"spinlock_wait_cycles":167543168,"thread_start_us":424862,"threads_started":2,"wal-append.queue_time_us":450046}
I20260808 01:11:05.801676 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:05.800311 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:05.804845 1603 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:05.806587 1490 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:05.809554 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:05.707010 4192 consensus_queue.cc:1046] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.080s
I20260808 01:11:05.699126 4249 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:05.840610 2202 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:05.842448 4197 raft_consensus.cc:2726] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:05.846594 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:05.857482 1408 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:05.805997 1410 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:05.716642 3801 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:05.785032 3689 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:05.861320 4336 log.cc:924] Time spent T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260808 01:11:05.867316 4087 raft_consensus.cc:2781] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:05.867775 4087 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:05.868240 4087 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:11:05.868671 4342 log.cc:924] Time spent T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20260808 01:11:05.863484 4184 raft_consensus.cc:2781] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:05.883291 4184 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:05.883718 4184 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:05.870020 2202 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260808 01:11:05.870580 1447 raft_consensus.cc:2454] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:11:05.886348 3772 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20260808 01:11:05.865252 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 59389us
I20260808 01:11:05.841017 1490 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:05.891070 1410 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:05.892725 4057 consensus_queue.cc:1035] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:11:05.894553 1565 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:05.897706 4331 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:05.898439 4331 consensus_queue.cc:1225] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer 95b661b03f1d460b9657e61d4b0247cd log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260808 01:11:05.898440 4251 consensus_queue.cc:1035] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:11:05.897703 4199 raft_consensus.cc:2726] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:05.900125 3772 raft_consensus.cc:2726] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:05.900594 4249 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:05.901070 4197 consensus_queue.cc:1035] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:11:05.902470 4197 consensus_queue.cc:1035] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:11:05.903029 4352 raft_consensus.cc:2741] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 01:11:05.844686 4166 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:05.902470 1565 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:05.904209 4166 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:11:05.903775 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 62838.9us
I20260808 01:11:05.906868 4166 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:05.907262 4326 log.cc:924] Time spent T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.567s user 0.002s sys 0.000s
I20260808 01:11:05.909371 4199 raft_consensus.cc:2781] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:05.909768 4199 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:05.910131 4199 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:05.911785 3801 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:05.721316 4189 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:05.855803 3992 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:05.918581 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:05.919930 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:05.869874 4099 raft_consensus.cc:3032] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:11:05.930512 4131 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:11:05.930832 4340 log.cc:924] Time spent T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.268s user 0.001s sys 0.000s
I20260808 01:11:05.933634 2202 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:48980
I20260808 01:11:05.940346 2202 raft_consensus.cc:595] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:05.934542 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:05.937987 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:05.914920 4289 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:05.935804 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:05.940999 2202 raft_consensus.cc:603] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:11:05.844868 4295 consensus_queue.cc:1046] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20260808 01:11:05.961484 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:05.966153 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:05.891520 4184 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:05.932211 4099 raft_consensus.cc:729] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:05.979785 3907 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
W20260808 01:11:06.044880 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 70455.5us
I20260808 01:11:05.856820 4291 consensus_queue.cc:237] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:05.983542 4127 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:06.057900 4099 consensus_queue.cc:260] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:06.053695 3907 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:06.062459 1445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51830
W20260808 01:11:06.049438 4340 log.cc:924] Time spent T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20260808 01:11:06.069463 4189 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:06.071906 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 169512us
I20260808 01:11:06.074414 1440 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:11:06.075109 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:06.075783 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:06.044811 4295 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:06.083547 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:06.085587 1410 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:06.139628 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 48829.3us
I20260808 01:11:06.149641 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:06.153172 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 67739us
I20260808 01:11:06.141562 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:05.964890 4289 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:06.049907 4040 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:06.089118 1599 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:06.155578 1565 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:06.063517 3907 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:06.151628 4337 raft_consensus.cc:1050] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
W20260808 01:11:06.157151 4115 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.503s user 0.016s sys 0.015s
I20260808 01:11:06.175438 1410 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:06.186600 1565 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:06.187961 1410 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:06.161722 3748 consensus_queue.cc:1035] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
W20260808 01:11:06.212462 4325 log.cc:924] Time spent T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.767s user 0.005s sys 0.015s
I20260808 01:11:06.164121 3915 raft_consensus.cc:2726] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:11:06.242223 4350 log.cc:924] Time spent T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.386s user 0.000s sys 0.002s
I20260808 01:11:06.238842 4041 raft_consensus.cc:2726] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:11:06.211396 4283 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.004s sys 0.033s
I20260808 01:11:06.190261 4273 raft_consensus.cc:2781] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:06.190374 1410 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:06.181932 3907 raft_consensus.cc:2726] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:06.141039 4295 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:06.165311 4256 consensus_queue.cc:1046] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20260808 01:11:06.161998 4269 consensus_queue.cc:1035] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
W20260808 01:11:06.250137 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 63654us
I20260808 01:11:06.278108 2201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 614) took 1172 ms. Trace:
I20260808 01:11:06.174396 4040 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:06.276463 4192 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:06.202222 4057 consensus_queue.cc:1035] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20260808 01:11:06.286288 4184 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:06.179190 4087 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:06.381904 4087 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:06.309247 4343 log.cc:924] Time spent T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
I20260808 01:11:06.249226 4189 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:06.363008 4273 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:06.403059 4273 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:06.402359 4189 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:06.313198 4337 consensus_queue.cc:237] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:06.315706 4115 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:06.412350 4115 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:06.414505 4041 raft_consensus.cc:3032] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:06.415184 4041 raft_consensus.cc:729] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:06.316593 4289 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:06.327453 2209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 779) took 1103 ms. Trace:
I20260808 01:11:06.329967 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:06.416428 4166 raft_consensus.cc:3032] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Stepping down as leader of term 3
W20260808 01:11:06.361455 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 171171us
I20260808 01:11:06.374903 2201 rpcz_store.cc:276] 0808 01:11:05.106055 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:05.168889 (+ 62834us) service_pool.cc:227] Handling call
0808 01:11:05.175990 (+ 7101us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:05.204083 (+ 28093us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:05.204115 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:05.204509 (+ 394us) log.cc:842] Serialized 26 byte log entry
0808 01:11:05.292329 (+ 87820us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:05.292380 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:05.292466 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:06.210214 (+917748us) spinlock_profiling.cc:243] Waited 6.04 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:06.245357 (+ 35143us) raft_consensus.cc:1671] finished
0808 01:11:06.245384 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:06.278013 (+ 32629us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13885312,"thread_start_us":86635,"threads_started":1,"wal-append.queue_time_us":115374}
I20260808 01:11:06.417225 4166 raft_consensus.cc:729] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:06.418555 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:06.419360 2202 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
W20260808 01:11:06.358532 4283 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s user 0.007s sys 0.033s
W20260808 01:11:06.384222 1340 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55758 (request call id 334) took 13356 ms (client timeout 15000 ms). Trace:
I20260808 01:11:06.205289 4259 consensus_queue.cc:1035] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.105s
I20260808 01:11:06.387107 1339 catalog_manager.cc:5495] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 2 to 3, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 3 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:11:06.416152 2209 rpcz_store.cc:276] 0808 01:11:05.224231 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:05.242005 (+ 17774us) service_pool.cc:227] Handling call
0808 01:11:05.242338 (+ 333us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:05.506588 (+264250us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:06.327198 (+820610us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b540004c754. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:06.327386 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":353623808}
I20260808 01:11:06.390834 4295 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:06.391253 4192 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:06.395025 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:06.397169 4179 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:06.363369 3915 raft_consensus.cc:2726] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:06.393658 4099 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:06.396857 4057 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:06.423738 4189 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:06.426184 1408 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:06.425509 1339 catalog_manager.cc:5495] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 1 to 2, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 2 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:11:06.389927 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:06.459045 2208 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:06.397405 4040 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:11:06.420328 1340 rpcz_store.cc:269] 0808 01:10:53.028235 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:53.032067 (+ 3832us) service_pool.cc:227] Handling call
0808 01:10:53.152897 (+120830us) spinlock_profiling.cc:243] Waited 42.3 ms on lock 0x7b5400002dd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4af0 00007fe34ca47f2a 00007fe352999d6c 00007fe352a93c3e 00007fe350da5fb4 00007fe350da5f42 00007fe350da5e71 00007fe350da5dec 00007fe350da50a2 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:06.384152 (+13231255us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 01:11:03.663170 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:11:03.705171 (+ 42001us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:11:03.705230 (+ 59us) write_op.cc:435] Acquired schema lock
0808 01:11:03.705243 (+ 13us) tablet.cc:660] Decoding operations
0808 01:11:03.705808 (+ 565us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:11:03.705935 (+ 127us) write_op.cc:641] Partition lock acquired for write op
0808 01:11:03.705985 (+ 50us) tablet.cc:683] Acquiring locks for 2 operations
0808 01:11:03.809519 (+103534us) tablet.cc:699] Row locks acquired
0808 01:11:03.818865 (+ 9346us) write_op.cc:260] PREPARE: finished
0808 01:11:03.850815 (+ 31950us) write_op.cc:270] Start()
0808 01:11:03.877284 (+ 26469us) write_op.cc:276] Timestamp: P: 1786151463843156 usec, L: 0
0808 01:11:03.882468 (+ 5184us) op_driver.cc:331] REPLICATION: starting
0808 01:11:04.054895 (+172427us) log.cc:842] Serialized 929 byte log entry
0808 01:11:04.798754 (+743859us) op_driver.cc:444] REPLICATION: finished
0808 01:11:05.174804 (+376050us) write_op.cc:301] APPLY: starting
0808 01:11:05.191624 (+ 16820us) tablet.cc:1365] starting BulkCheckPresence
0808 01:11:05.249127 (+ 57503us) tablet.cc:1368] finished BulkCheckPresence
0808 01:11:05.297174 (+ 48047us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 01:11:05.420713 (+123539us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 01:11:05.421060 (+ 347us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0808 01:11:05.421113 (+ 53us) write_op.cc:312] APPLY: finished
0808 01:11:05.421449 (+ 336us) log.cc:842] Serialized 29 byte log entry
0808 01:11:05.421661 (+ 212us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:11:05.421764 (+ 103us) write_op.cc:454] Released schema lock
0808 01:11:05.421827 (+ 63us) write_op.cc:341] FINISH: Updating metrics
0808 01:11:05.422197 (+ 370us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:11:05.422230 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2535988,"reported_tablets":6,"spinlock_wait_cycles":97341184,"child_traces":[["op",{"apply.queue_time_us":320648,"apply.run_cpu_time_us":3508,"apply.run_wall_time_us":258808,"num_ops":2,"prepare.queue_time_us":223970,"prepare.run_cpu_time_us":30283,"prepare.run_wall_time_us":712233,"raft.queue_time_us":262084,"raft.run_cpu_time_us":10085,"raft.run_wall_time_us":135949,"replication_time_us":909331,"thread_start_us":488409,"threads_started":4,"wal-append.queue_time_us":412940}]]}
I20260808 01:11:06.429977 1448 raft_consensus.cc:1226] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260808 01:11:06.460294 4057 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:06.460700 4087 consensus_queue.cc:1046] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:11:06.462706 4087 raft_consensus.cc:3032] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:11:06.432554 1599 raft_consensus.cc:1226] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 01:11:06.434501 1410 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:06.436829 1602 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260808 01:11:06.446794 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:06.465319 1604 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:06.466109 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:06.466845 1597 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:06.467694 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:06.468411 1600 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:06.449784 1490 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:06.418913 4166 consensus_queue.cc:260] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:06.452672 2205 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.797s user 0.065s sys 0.072s
I20260808 01:11:06.475209 1604 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 8.
W20260808 01:11:06.475608 1444 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.826s user 0.013s sys 0.022s
I20260808 01:11:06.438695 4352 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:11:06.480401 2205 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.824s user 0.065s sys 0.072s
W20260808 01:11:06.481813 1444 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.832s user 0.013s sys 0.022s
W20260808 01:11:06.483904 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 49514.8us
I20260808 01:11:06.497973 1410 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:06.479601 1600 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:06.463261 4057 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:06.503913 1410 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:06.521117 4333 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.818s user 0.026s sys 0.075s
W20260808 01:11:06.522004 4333 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.823s user 0.026s sys 0.076s
I20260808 01:11:06.456182 2203 raft_consensus.cc:2454] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:06.472757 3915 raft_consensus.cc:2781] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:06.533489 1410 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:06.485940 1594 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.720s user 0.014s sys 0.028s
I20260808 01:11:06.463356 4087 raft_consensus.cc:729] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
W20260808 01:11:06.561607 1594 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.796s user 0.014s sys 0.028s
W20260808 01:11:06.569695 1525 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.779s user 0.017s sys 0.019s
W20260808 01:11:06.570199 1525 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.779s user 0.018s sys 0.020s
W20260808 01:11:06.481868 1568 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.798s user 0.016s sys 0.033s
W20260808 01:11:06.574055 1568 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.890s user 0.016s sys 0.033s
W20260808 01:11:06.574399 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 897946us
I20260808 01:11:06.573511 2208 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:06.580541 1408 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:06.522495 4333 raft_consensus.cc:686] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
W20260808 01:11:06.571198 1520 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.009s sys 0.027s
I20260808 01:11:06.480222 4317 raft_consensus.cc:2781] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:06.455952 4179 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:06.499296 4274 raft_consensus.cc:2726] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:06.490331 4115 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:06.490235 4189 raft_consensus.cc:2726] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:06.509853 4166 raft_consensus.cc:3037] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:06.531893 4295 raft_consensus.cc:2781] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 01:11:06.592941 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 59563.7us
I20260808 01:11:06.578394 3915 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:06.576357 4127 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:06.552062 4251 raft_consensus.cc:2726] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:06.595897 1408 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
W20260808 01:11:06.594467 1520 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.725s user 0.010s sys 0.027s
I20260808 01:11:06.575878 4289 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:06.501871 4352 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:06.593782 4317 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:06.576910 3907 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
I20260808 01:11:06.606293 3772 raft_consensus.cc:2781] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election won for term 7
W20260808 01:11:06.601553 4113 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.996s user 0.004s sys 0.015s
I20260808 01:11:06.598523 4134 raft_consensus.cc:2781] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader election won for term 8
W20260808 01:11:06.723877 1519 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.011s sys 0.041s
W20260808 01:11:06.825610 1519 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.775s user 0.012s sys 0.041s
I20260808 01:11:06.602630 4295 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:11:06.741593 1523 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.007s sys 0.027s
W20260808 01:11:06.829269 1523 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.759s user 0.013s sys 0.029s
W20260808 01:11:06.725646 1487 leader_election.cc:334] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.604s (SENT)
I20260808 01:11:06.830119 1487 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:06.831153 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 105565us
W20260808 01:11:06.832298 1487 leader_election.cc:334] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.612s (SENT)
I20260808 01:11:06.832769 1487 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:06.480785 4041 consensus_queue.cc:260] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:06.614264 4009 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:06.614825 3915 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:06.708889 4189 raft_consensus.cc:2781] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:06.617063 4127 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:06.834844 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:06.732839 4317 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:11:06.727972 4199 consensus_meta.cc:287] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.817s user 0.014s sys 0.007s
W20260808 01:11:06.825855 4113 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.220s user 0.006s sys 0.016s
I20260808 01:11:06.797751 4197 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:06.834844 4275 raft_consensus.cc:2726] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:06.849189 4319 raft_consensus.cc:2726] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:06.848875 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:06.849954 1599 raft_consensus.cc:1261] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:11:06.898087 1522 raft_consensus.cc:1226] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 01:11:06.851852 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:06.859496 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:06.900120 1604 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:06.900801 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:06.867391 4189 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:06.906958 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:06.815802 4251 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:06.906209 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:06.850567 1598 raft_consensus.cc:1226] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:06.834859 4252 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:06.855435 4358 log.cc:924] Time spent T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
I20260808 01:11:06.857120 4009 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:06.822477 4289 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:06.834932 4295 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:06.858040 4134 raft_consensus.cc:686] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:06.846887 4357 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:06.874782 4115 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:06.907106 4041 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:06.915578 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:06.952085 1604 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 7.
W20260808 01:11:06.955633 1516 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.984s user 0.023s sys 0.047s
W20260808 01:11:06.975275 1516 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.003s user 0.024s sys 0.047s
W20260808 01:11:06.903385 4291 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.661s user 0.012s sys 0.038s
I20260808 01:11:06.839198 4333 consensus_queue.cc:237] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:06.873654 4146 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:06.909718 4189 raft_consensus.cc:3037] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:06.924377 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:06.953182 4179 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:06.903165 4197 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:06.918296 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:06.977936 1522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2111) took 1280 ms. Trace:
I20260808 01:11:06.986318 1522 rpcz_store.cc:276] 0808 01:11:05.697949 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:05.698194 (+ 245us) service_pool.cc:227] Handling call
0808 01:11:05.698550 (+ 356us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:06.906150 (+1207600us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:06.977620 (+ 71470us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540012f054. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:06.977866 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2758006016}
I20260808 01:11:06.925419 4192 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:06.977936 1446 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.042s user 0.035s sys 0.056s
I20260808 01:11:06.955495 4356 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:06.989114 1446 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.053s user 0.036s sys 0.056s
I20260808 01:11:06.896997 4199 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:06.977674 4291 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s user 0.012s sys 0.039s
I20260808 01:11:06.961761 4252 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:07.005792 2172 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.211s user 0.022s sys 0.044s
W20260808 01:11:07.006173 2172 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.220s user 0.022s sys 0.045s
I20260808 01:11:06.914918 4251 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:07.005739 1443 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.092s user 0.029s sys 0.028s
W20260808 01:11:07.013123 1443 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.099s user 0.029s sys 0.028s
I20260808 01:11:06.989318 4259 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:07.039225 1519 raft_consensus.cc:1261] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 01:11:06.988324 4146 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:07.048187 4252 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:07.048640 4146 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:07.049393 4127 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:07.009616 4368 log.cc:924] Time spent T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.114s user 0.000s sys 0.001s
I20260808 01:11:06.974917 4115 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:07.060156 2205 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.009s sys 0.048s
W20260808 01:11:07.064989 2205 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s user 0.010s sys 0.048s
I20260808 01:11:07.065722 2205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 786) took 1439 ms. Trace:
W20260808 01:11:07.066915 2200 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.086s user 0.036s sys 0.050s
W20260808 01:11:07.067373 2200 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.086s user 0.036s sys 0.050s
W20260808 01:11:07.071007 2171 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.265s user 0.027s sys 0.046s
W20260808 01:11:07.105173 2171 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.299s user 0.027s sys 0.047s
I20260808 01:11:06.965302 4357 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:07.012005 4356 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:07.105571 2171 connection.cc:232] RPC call timeout handler was delayed by -1.5836685439862777s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1622s for a response
I20260808 01:11:07.124235 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:07.066082 2205 rpcz_store.cc:276] 0808 01:11:05.626484 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:05.626690 (+ 206us) service_pool.cc:227] Handling call
0808 01:11:05.649976 (+ 23286us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:05.650265 (+ 289us) raft_consensus.cc:1520] Early marking committed up to index 10
0808 01:11:06.495451 (+845186us) log.cc:842] Serialized 14 byte log entry
0808 01:11:07.065390 (+569939us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:07.065443 (+ 53us) raft_consensus.cc:1631] Marking committed up to 10
0808 01:11:07.065491 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:07.065554 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:07.065669 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":117422,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"thread_start_us":1408909,"threads_started":2}
I20260808 01:11:07.127154 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:07.127456 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:07.127760 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:07.128013 2202 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:07.058768 4291 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:07.145458 4357 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:07.146884 1520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1965) took 1378 ms. Trace:
I20260808 01:11:07.071043 4113 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:07.153246 4099 consensus_meta.cc:287] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.695s user 0.005s sys 0.006s
W20260808 01:11:07.153506 4359 log.cc:924] Time spent T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.276s user 0.000s sys 0.004s
I20260808 01:11:07.154106 4009 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:07.155268 4197 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:07.061249 4259 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:07.155467 4370 log.cc:924] Time spent T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
W20260808 01:11:07.150445 4364 log.cc:924] Time spent T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.273s user 0.002s sys 0.002s
I20260808 01:11:07.105018 4352 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:07.164754 1518 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.002s sys 0.030s
W20260808 01:11:07.134189 4358 log.cc:924] Time spent T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260808 01:11:07.161129 1520 rpcz_store.cc:276] 0808 01:11:05.768320 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:05.769024 (+ 704us) service_pool.cc:227] Handling call
0808 01:11:05.769444 (+ 420us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:05.790091 (+ 20647us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 01:11:05.790134 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:05.790597 (+ 463us) log.cc:842] Serialized 26 byte log entry
0808 01:11:06.328397 (+537800us) spinlock_profiling.cc:243] Waited 27.7 ms on lock 0x7b4800250758. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3518e67f1 00007fe3518dd8e7 00007fe3518dd77a 00007fe3519d195f 00007fe3519d1627 00007fe3519e5b2c 00007fe3519e5ad8 00007fe3519e5a58 00007fe3519e5a13 00007fe3519e4cbc 00007fe350966be6
0808 01:11:06.618560 (+290163us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:06.618631 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:06.618763 (+ 132us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:07.146558 (+527795us) raft_consensus.cc:1671] finished
0808 01:11:07.146581 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:07.146807 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20208,"raft.queue_time_us":34640,"raft.run_cpu_time_us":16565,"raft.run_wall_time_us":514571,"spinlock_wait_cycles":63639808,"thread_start_us":749202,"threads_started":1,"wal-append.queue_time_us":988990}
I20260808 01:11:07.166792 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:07.167045 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:07.122685 4252 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:07.170787 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:07.164281 4291 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:07.190281 1524 raft_consensus.cc:1261] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260808 01:11:07.192680 2201 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:07.195207 2207 raft_consensus.cc:2379] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 4: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
W20260808 01:11:07.195577 1518 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s user 0.002s sys 0.030s
W20260808 01:11:07.193806 1411 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.027s sys 0.048s
W20260808 01:11:07.199455 1411 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.996s user 0.028s sys 0.048s
I20260808 01:11:07.072008 4087 consensus_queue.cc:260] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:07.065122 4134 consensus_queue.cc:237] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:07.217082 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:07.223685 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:07.224361 2209 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:07.226822 1521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2108) took 1587 ms. Trace:
I20260808 01:11:07.227262 1521 rpcz_store.cc:276] 0808 01:11:05.639339 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:05.639485 (+ 146us) service_pool.cc:227] Handling call
0808 01:11:05.653352 (+ 13867us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:05.653742 (+ 390us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:05.653770 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 01:11:05.669727 (+ 15957us) log.cc:842] Serialized 52 byte log entry
0808 01:11:06.183787 (+514060us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:06.287047 (+103260us) log.cc:842] Serialized 14 byte log entry
0808 01:11:06.287299 (+ 252us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:06.287399 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:07.226581 (+939182us) raft_consensus.cc:1671] finished
0808 01:11:07.226612 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:07.226758 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4189,"prepare.queue_time_us":104320,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":75,"raft.queue_time_us":15065,"raft.run_cpu_time_us":375,"raft.run_wall_time_us":21137,"thread_start_us":454172,"threads_started":1,"wal-append.queue_time_us":855683}
I20260808 01:11:07.202302 4234 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:07.208544 4146 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:07.273800 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 441495us
I20260808 01:11:07.166882 4113 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:07.216387 4289 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:07.306856 1444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2079) took 1817 ms. Trace:
I20260808 01:11:07.307331 1444 rpcz_store.cc:276] 0808 01:11:05.489948 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:05.496979 (+ 7031us) service_pool.cc:227] Handling call
0808 01:11:05.497380 (+ 401us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:05.593986 (+ 96606us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:05.594033 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:05.620418 (+ 26385us) log.cc:842] Serialized 26 byte log entry
0808 01:11:06.496619 (+876201us) raft_consensus.cc:1631] Marking committed up to 3
0808 01:11:06.496675 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:06.496892 (+ 217us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:07.149790 (+652898us) spinlock_profiling.cc:243] Waited 2.76 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:07.298633 (+148843us) raft_consensus.cc:1671] finished
0808 01:11:07.298657 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:07.306787 (+ 8130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":74594,"raft.queue_time_us":93787,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":2997,"spinlock_wait_cycles":6341376,"thread_start_us":847095,"threads_started":1,"wal-append.queue_time_us":974796}
W20260808 01:11:07.301707 4380 log.cc:924] Time spent T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.104s user 0.000s sys 0.001s
I20260808 01:11:07.340526 4369 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:07.341349 4369 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:07.344257 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:07.344933 4369 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:07.345351 2207 raft_consensus.cc:2379] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 6: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:07.345700 4369 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:07.346372 4369 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:07.347007 4376 log.cc:924] Time spent T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.155s user 0.001s sys 0.000s
I20260808 01:11:07.274398 4199 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:07.352669 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:07.355273 4291 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:07.357295 4259 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:07.357597 1523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1970) took 1418 ms. Trace:
I20260808 01:11:07.307047 4099 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:07.358536 4099 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:07.366818 4379 log.cc:924] Time spent T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260808 01:11:07.372166 1594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2223) took 1686 ms. Trace:
W20260808 01:11:07.372763 3915 consensus_meta.cc:287] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.503s user 0.005s sys 0.006s
I20260808 01:11:07.358171 1523 rpcz_store.cc:276] 0808 01:11:05.939249 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:05.939381 (+ 132us) service_pool.cc:227] Handling call
0808 01:11:05.968318 (+ 28937us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:05.968693 (+ 375us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:05.968723 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 01:11:05.975963 (+ 7240us) log.cc:842] Serialized 52 byte log entry
0808 01:11:06.829736 (+853773us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:06.829777 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:06.829860 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:07.357282 (+527422us) raft_consensus.cc:1671] finished
0808 01:11:07.357310 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:07.357519 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2139,"raft.queue_time_us":2776,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":6877,"thread_start_us":759247,"threads_started":1,"wal-append.queue_time_us":1069355}
I20260808 01:11:07.378875 2201 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:07.214531 4252 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:07.372761 1594 rpcz_store.cc:276] 0808 01:11:05.685928 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:05.686114 (+ 186us) service_pool.cc:227] Handling call
0808 01:11:05.686568 (+ 454us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:05.760260 (+ 73692us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:05.760298 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:05.760747 (+ 449us) log.cc:842] Serialized 26 byte log entry
0808 01:11:06.562125 (+801378us) raft_consensus.cc:1631] Marking committed up to 2
0808 01:11:06.562179 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:06.562270 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:07.152905 (+590635us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:07.335237 (+182332us) raft_consensus.cc:1671] finished
0808 01:11:07.335267 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:07.372071 (+ 36804us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":16355,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":3611,"spinlock_wait_cycles":6519680,"thread_start_us":861110,"threads_started":2,"wal-append.queue_time_us":950379}
W20260808 01:11:07.396891 1410 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.022s sys 0.015s
W20260808 01:11:07.397378 1410 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.590s user 0.022s sys 0.015s
I20260808 01:11:07.397281 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:07.398494 1447 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:07.402660 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:07.403508 1447 raft_consensus.cc:2454] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:07.415573 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:07.415757 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:07.416743 4166 consensus_meta.cc:287] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.799s user 0.012s sys 0.001s
I20260808 01:11:07.303293 4146 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:07.417373 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:07.418087 4099 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:07.419423 4146 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:07.300307 4234 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:07.420007 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:07.420282 4317 consensus_meta.cc:287] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.512s user 0.011s sys 0.000s
I20260808 01:11:07.421366 2208 pending_rounds.cc:76] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd: Aborting all ops after (but not including) 6
I20260808 01:11:07.474493 2208 pending_rounds.cc:97] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260808 01:11:07.474957 2208 raft_consensus.cc:2866] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:07.474511 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:07.421779 4252 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:07.422588 4356 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:07.423511 2209 raft_consensus.cc:1261] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:11:07.424388 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:07.419813 2171 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:07.435070 4115 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:07.437953 4369 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:07.442916 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:07.336396 4251 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:07.513432 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:07.448562 1408 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.639s user 0.017s sys 0.065s
I20260808 01:11:07.450708 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:07.515673 1408 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.706s user 0.017s sys 0.066s
I20260808 01:11:07.519986 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:07.468137 1490 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.597s user 0.017s sys 0.017s
I20260808 01:11:07.475859 1523 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
W20260808 01:11:07.489199 1487 leader_election.cc:334] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.902s (SENT)
I20260808 01:11:07.505517 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:07.507223 2203 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
W20260808 01:11:07.454704 4373 log.cc:924] Time spent T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.257s user 0.001s sys 0.001s
I20260808 01:11:07.396423 3915 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:07.524468 1487 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:07.536377 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 47241.5us
I20260808 01:11:07.538268 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:07.529203 1516 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.019s sys 0.032s
I20260808 01:11:07.520740 1521 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:07.539548 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:07.540069 1516 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.551s user 0.020s sys 0.032s
I20260808 01:11:07.530148 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:07.531104 1446 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.010s sys 0.025s
I20260808 01:11:07.523487 1599 raft_consensus.cc:2454] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
W20260808 01:11:07.572001 1446 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.571s user 0.013s sys 0.025s
W20260808 01:11:07.513620 4165 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.157s user 0.024s sys 0.030s
I20260808 01:11:07.569190 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:07.539157 1523 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:07.540278 2206 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:07.579013 2204 raft_consensus.cc:2362] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5. Re-sending same reply.
W20260808 01:11:07.525264 1490 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.654s user 0.018s sys 0.017s
I20260808 01:11:07.571269 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:07.506503 4099 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:07.573483 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:07.576131 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:07.577267 1516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2117) took 1644 ms. Trace:
I20260808 01:11:07.573263 1446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55042 (request call id 2368) took 1778 ms. Trace:
W20260808 01:11:07.514787 4353 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.013s sys 0.044s
W20260808 01:11:07.573055 1487 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.742s (SENT) [suppressed 44 similar messages]
I20260808 01:11:07.520327 4009 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:07.614032 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:07.615002 1601 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:07.541332 4251 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:07.594803 1446 rpcz_store.cc:276] 0808 01:11:05.794945 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:05.810610 (+ 15665us) service_pool.cc:227] Handling call
0808 01:11:05.851142 (+ 40532us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:05.861583 (+ 10441us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:05.861627 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:05.862383 (+ 756us) log.cc:842] Serialized 26 byte log entry
0808 01:11:06.989557 (+1127174us) raft_consensus.cc:1631] Marking committed up to 3
0808 01:11:07.572563 (+583006us) log.cc:842] Serialized 14 byte log entry
0808 01:11:07.572865 (+ 302us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:07.572934 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:07.573073 (+ 139us) raft_consensus.cc:1671] finished
0808 01:11:07.573088 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:07.573211 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9917,"prepare.queue_time_us":225107,"raft.queue_time_us":11795,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"spinlock_wait_cycles":22048512,"thread_start_us":1624736,"threads_started":2,"wal-append.queue_time_us":1321194}
I20260808 01:11:07.505143 4317 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:07.623054 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:11:07.597543 4353 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.692s user 0.013s sys 0.045s
I20260808 01:11:07.575941 4274 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:07.539368 4166 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:07.660065 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 74647.1us
I20260808 01:11:07.571141 4197 raft_consensus.cc:2726] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:07.629014 2200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 792) took 1740 ms. Trace:
I20260808 01:11:07.662123 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:07.578989 4165 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.222s user 0.024s sys 0.031s
I20260808 01:11:07.650782 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:07.645056 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:07.659396 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:07.662722 2200 rpcz_store.cc:276] 0808 01:11:05.888602 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:05.888749 (+ 147us) service_pool.cc:227] Handling call
0808 01:11:05.889128 (+ 379us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:05.924326 (+ 35198us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:05.924371 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 01:11:05.937134 (+ 12763us) log.cc:842] Serialized 52 byte log entry
0808 01:11:07.102498 (+1165364us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:07.628272 (+525774us) log.cc:842] Serialized 14 byte log entry
0808 01:11:07.628576 (+ 304us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:07.628680 (+ 104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:07.628737 (+ 57us) raft_consensus.cc:1671] finished
0808 01:11:07.628757 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:07.628942 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6595,"raft.queue_time_us":36637,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":6677,"thread_start_us":1580023,"threads_started":2,"wal-append.queue_time_us":1265542}
I20260808 01:11:07.611639 4291 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:07.693411 1523 raft_consensus.cc:1226] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:07.696056 3748 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:07.696482 2200 raft_consensus.cc:1226] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 1. Current term is 2. Ops: []
I20260808 01:11:07.696655 3748 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:07.699128 3748 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:11:07.660035 4371 log.cc:924] Time spent T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260808 01:11:07.609381 4099 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:07.707944 1521 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
W20260808 01:11:07.688324 4382 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.016s sys 0.012s
W20260808 01:11:07.743758 1489 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.886s user 0.011s sys 0.033s
W20260808 01:11:07.688705 4379 log.cc:924] Time spent T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
W20260808 01:11:07.769961 4382 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s user 0.017s sys 0.012s
W20260808 01:11:07.773552 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 316406us
I20260808 01:11:07.761543 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:07.712040 4189 consensus_meta.cc:287] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.709s user 0.017s sys 0.003s
W20260808 01:11:07.770231 1489 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.913s user 0.011s sys 0.033s
I20260808 01:11:07.692488 4386 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:07.705847 4113 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:07.778502 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 920927us
I20260808 01:11:07.758704 4197 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:07.623692 4009 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:07.776350 1565 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.907s user 0.014s sys 0.026s
W20260808 01:11:07.797489 1565 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.929s user 0.014s sys 0.026s
I20260808 01:11:07.687314 4166 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:07.763202 4273 consensus_meta.cc:287] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.359s user 0.011s sys 0.006s
I20260808 01:11:07.779094 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:07.779577 1568 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:11:07.766260 4377 log.cc:924] Time spent T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260808 01:11:07.787154 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:07.804760 2204 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
W20260808 01:11:07.738668 1596 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.847s user 0.038s sys 0.062s
I20260808 01:11:07.762722 4369 raft_consensus.cc:2781] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:07.663411 4251 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:07.662089 4274 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:07.802263 1565 leader_election.cc:334] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.616s (SENT)
I20260808 01:11:07.800618 2209 raft_consensus.cc:2454] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
W20260808 01:11:07.805872 1596 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.925s user 0.039s sys 0.062s
I20260808 01:11:07.741760 4291 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:07.763296 4115 consensus_queue.cc:1035] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
W20260808 01:11:07.801852 4363 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s user 0.009s sys 0.021s
W20260808 01:11:07.812407 1448 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.014s sys 0.026s
I20260808 01:11:07.821341 1568 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:11:07.823151 1448 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s user 0.014s sys 0.026s
I20260808 01:11:07.788157 4189 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:07.807065 4369 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:07.812655 4041 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
W20260808 01:11:07.841773 4363 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.670s user 0.010s sys 0.022s
I20260808 01:11:07.799151 4197 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:07.839702 4264 raft_consensus.cc:2741] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20260808 01:11:07.921754 2170 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.509s user 0.012s sys 0.020s
I20260808 01:11:07.793987 4386 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:07.923416 4166 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:07.803161 4273 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:07.844172 3915 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:07.849282 4369 raft_consensus.cc:3037] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:07.947932 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:07.853720 4113 consensus_queue.cc:1035] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.109s
W20260808 01:11:07.983429 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 384837us
I20260808 01:11:08.020628 1445 raft_consensus.cc:595] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:08.022737 1445 raft_consensus.cc:603] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:11:08.024194 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 451126us
W20260808 01:11:08.026425 2205 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.008s sys 0.020s
W20260808 01:11:08.028158 2205 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.578s user 0.008s sys 0.020s
I20260808 01:11:07.957720 4317 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:08.026852 2202 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.003s sys 0.025s
I20260808 01:11:08.030429 1597 raft_consensus.cc:1226] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:07.961033 4264 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:08.048801 1410 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:07.986693 4353 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:08.049700 2202 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s user 0.003s sys 0.025s
I20260808 01:11:07.898571 4041 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:08.054644 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:08.059279 2206 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:11:08.062044 1487 leader_election.cc:334] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.048s (SENT)
I20260808 01:11:08.023936 4249 raft_consensus.cc:2726] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:08.082499 1410 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:11:08.079912 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 279373us
W20260808 01:11:08.051721 2171 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.013s sys 0.030s
W20260808 01:11:08.086993 2171 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.577s user 0.014s sys 0.030s
W20260808 01:11:08.087518 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 667754us
I20260808 01:11:08.092037 2171 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
W20260808 01:11:08.065357 4388 log.cc:924] Time spent T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.321s user 0.000s sys 0.001s
W20260808 01:11:08.109902 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 93591.5us
W20260808 01:11:08.069332 3772 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.222s user 0.020s sys 0.017s
W20260808 01:11:08.112015 1565 leader_election.cc:334] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.348s (SENT)
W20260808 01:11:08.078919 4295 consensus_meta.cc:287] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 1.113s user 0.012s sys 0.002s
W20260808 01:11:08.139387 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 61677.9us
I20260808 01:11:07.926040 4392 raft_consensus.cc:2781] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:08.084323 4099 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:08.076550 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:08.148458 4416 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:08.158149 4234 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:08.166150 1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2556) took 1315 ms. Trace:
I20260808 01:11:08.323904 1597 rpcz_store.cc:276] 0808 01:11:06.850496 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:06.852900 (+ 2404us) service_pool.cc:227] Handling call
0808 01:11:06.893690 (+ 40790us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:08.049267 (+1155577us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:08.165810 (+116543us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54000550d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:08.166064 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2613643264}
I20260808 01:11:08.168465 1524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1972) took 2091 ms. Trace:
W20260808 01:11:08.173275 4361 log.cc:924] Time spent T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 1.295s user 0.012s sys 0.000s
I20260808 01:11:08.181394 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:08.208552 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 112992us
W20260808 01:11:08.104558 4389 log.cc:924] Time spent T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.341s user 0.000s sys 0.001s
W20260808 01:11:08.123081 3772 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.280s user 0.020s sys 0.017s
W20260808 01:11:08.292634 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 180650us
I20260808 01:11:08.112720 4165 raft_consensus.cc:2781] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:08.347092 3772 raft_consensus.cc:686] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:08.347360 4165 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:08.347914 4165 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:08.348407 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:08.348078 3772 consensus_queue.cc:237] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.349032 2207 raft_consensus.cc:2454] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:08.349964 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:08.222472 4392 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:08.351951 2209 raft_consensus.cc:3032] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:11:08.352188 4392 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:08.352417 2209 raft_consensus.cc:729] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:08.353195 2209 consensus_queue.cc:260] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.354177 2209 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:08.340536 4189 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:08.356107 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:08.356242 1525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2112) took 2657 ms. Trace:
I20260808 01:11:08.301918 4009 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:08.356729 1525 rpcz_store.cc:276] 0808 01:11:05.698364 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:05.699303 (+ 939us) service_pool.cc:227] Handling call
0808 01:11:05.716687 (+ 17384us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:06.570721 (+854034us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:06.570751 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 01:11:06.608154 (+ 37403us) log.cc:842] Serialized 52 byte log entry
0808 01:11:07.202152 (+593998us) raft_consensus.cc:1631] Marking committed up to 8
0808 01:11:07.202185 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:07.202997 (+ 812us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:08.355973 (+1152976us) raft_consensus.cc:1671] finished
0808 01:11:08.356000 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:08.356166 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51978,"raft.queue_time_us":52428,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":25138,"thread_start_us":1152461,"threads_started":2,"wal-append.queue_time_us":898802}
W20260808 01:11:08.127063 4384 log.cc:924] Time spent T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.338s user 0.001s sys 0.001s
I20260808 01:11:08.055398 4264 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.303475 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:08.064919 4353 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.366631 1567 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:08.367190 1440 raft_consensus.cc:1261] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260808 01:11:08.367769 1440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2092) took 1025 ms. Trace:
I20260808 01:11:08.368121 1440 rpcz_store.cc:276] 0808 01:11:07.342429 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:07.342557 (+ 128us) service_pool.cc:227] Handling call
0808 01:11:08.366706 (+1024149us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b5800003698. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adf851 00007fe349adf702 00007fe3529cfb09 00007fe35190b730 00007fe3519d179b 00007fe3519d1821 00007fe3519952b0 00007fe3519807f9 00007fe3519802a2 00007fe34b066684
0808 01:11:08.366905 (+ 199us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:08.367548 (+ 643us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:08.367720 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41158016}
I20260808 01:11:08.368695 4057 raft_consensus.cc:2781] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:08.369122 4057 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:08.369289 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:08.337980 1487 leader_election.cc:334] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.983s (SENT)
I20260808 01:11:08.300874 4098 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:08.369920 1440 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:08.369953 1487 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:08.370162 4098 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.370903 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:08.371402 1487 leader_election.cc:334] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.545s (SENT)
I20260808 01:11:08.371527 1440 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:08.346567 4099 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:08.359577 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:08.360085 1438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55042 (request call id 2373) took 1941 ms. Trace:
I20260808 01:11:08.360826 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:08.363322 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:08.446342 2200 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:08.369470 4057 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:08.371820 1487 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:08.450946 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 79571.7us
W20260808 01:11:08.451598 1487 leader_election.cc:334] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.804s (SENT)
I20260808 01:11:08.372354 4275 raft_consensus.cc:2726] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:08.372519 4098 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:08.343274 1442 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:08.372711 1443 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55042 (request call id 2369) took 2573 ms. Trace:
I20260808 01:11:08.374639 1518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1975) took 2056 ms. Trace:
I20260808 01:11:08.367542 1567 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:08.378245 4165 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.388335 2206 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:08.460031 1518 rpcz_store.cc:276] 0808 01:11:06.318392 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:06.321484 (+ 3092us) service_pool.cc:227] Handling call
0808 01:11:06.321894 (+ 410us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:06.489154 (+167260us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:06.489212 (+ 58us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 01:11:06.562562 (+ 73350us) log.cc:842] Serialized 52 byte log entry
0808 01:11:07.198907 (+636345us) raft_consensus.cc:1631] Marking committed up to 1
0808 01:11:07.198962 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:08.374241 (+1175279us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54000df554. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:08.374292 (+ 51us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:08.374340 (+ 48us) raft_consensus.cc:1671] finished
0808 01:11:08.374354 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:08.374577 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7532,"raft.queue_time_us":7645,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":23944,"spinlock_wait_cycles":347640576,"thread_start_us":603566,"threads_started":1,"wal-append.queue_time_us":1103892}
I20260808 01:11:08.394404 1448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2086) took 1189 ms. Trace:
W20260808 01:11:08.304770 4404 log.cc:924] Time spent T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
W20260808 01:11:08.396174 4408 log.cc:924] Time spent T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.173s user 0.000s sys 0.001s
W20260808 01:11:08.396875 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 314458us
I20260808 01:11:08.325555 1524 rpcz_store.cc:276] 0808 01:11:06.077023 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:06.077231 (+ 208us) service_pool.cc:227] Handling call
0808 01:11:06.077678 (+ 447us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:07.198050 (+1120372us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:08.168170 (+970120us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540012eb54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:08.168390 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2556490624}
I20260808 01:11:08.269070 4394 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:08.439383 2209 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260808 01:11:08.293272 4291 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:08.305680 4416 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:08.458510 1443 rpcz_store.cc:276] 0808 01:11:05.799366 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:05.799954 (+ 588us) service_pool.cc:227] Handling call
0808 01:11:05.800422 (+ 468us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:05.912815 (+112393us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:05.912858 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:05.913359 (+ 501us) log.cc:842] Serialized 26 byte log entry
0808 01:11:07.013562 (+1100203us) raft_consensus.cc:1631] Marking committed up to 3
0808 01:11:07.537454 (+523892us) log.cc:842] Serialized 14 byte log entry
0808 01:11:08.020360 (+482906us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:08.372297 (+351937us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54001d2c54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:08.372342 (+ 45us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:08.372388 (+ 46us) raft_consensus.cc:1671] finished
0808 01:11:08.372492 (+ 104us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:08.372626 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":63932,"prepare.queue_time_us":767382,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":263666,"raft.run_cpu_time_us":356,"raft.run_wall_time_us":12242,"spinlock_wait_cycles":257520384,"thread_start_us":1946000,"threads_started":3,"wal-append.queue_time_us":1295566}
I20260808 01:11:08.467005 1448 rpcz_store.cc:276] 0808 01:11:07.204539 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:07.204726 (+ 187us) service_pool.cc:227] Handling call
0808 01:11:07.205070 (+ 344us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:07.205430 (+ 360us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:07.205460 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:07.205908 (+ 448us) log.cc:842] Serialized 26 byte log entry
0808 01:11:07.847674 (+641766us) raft_consensus.cc:1631] Marking committed up to 1
0808 01:11:07.847714 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:07.847792 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:08.394139 (+546347us) raft_consensus.cc:1671] finished
0808 01:11:08.394166 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:08.394326 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9750,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":23853,"thread_start_us":618444,"threads_started":1,"wal-append.queue_time_us":906025}
W20260808 01:11:08.465884 4388 log.cc:924] Time spent T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
I20260808 01:11:08.469827 4375 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:08.471093 4273 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:08.447434 4251 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:08.445767 1438 rpcz_store.cc:276] 0808 01:11:06.418672 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:06.424852 (+ 6180us) service_pool.cc:227] Handling call
0808 01:11:06.425555 (+ 703us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:07.504954 (+1079399us) raft_consensus.cc:1520] Early marking committed up to index 2
0808 01:11:07.507937 (+ 2983us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:07.511851 (+ 3914us) raft_consensus.cc:1631] Marking committed up to 2
0808 01:11:07.521907 (+ 10056us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:08.359746 (+837839us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540009db54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:08.359816 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:08.360003 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1838,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":42286,"spinlock_wait_cycles":2346080128}
I20260808 01:11:08.475934 1442 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55040 (request call id 2093) took 1115 ms. Trace:
I20260808 01:11:08.483297 4275 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:08.478564 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 111093us
I20260808 01:11:08.481438 1568 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:08.513046 1568 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:08.451156 4099 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.486176 2209 pending_rounds.cc:76] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: Aborting all ops after (but not including) 8
W20260808 01:11:08.461687 4439 log.cc:924] Time spent T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.090s user 0.002s sys 0.000s
I20260808 01:11:08.451352 3861 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:08.130550 4295 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.495580 4433 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:08.483879 4401 raft_consensus.cc:2726] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:08.483187 4255 consensus_queue.cc:1035] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.037s
I20260808 01:11:08.457880 4113 raft_consensus.cc:2726] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:08.339007 2203 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:08.491977 4416 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:08.488193 4293 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:08.517338 2209 pending_rounds.cc:97] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260808 01:11:08.532305 2209 raft_consensus.cc:2866] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:08.520407 1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2239) took 2075 ms. Trace:
I20260808 01:11:08.509982 1442 rpcz_store.cc:276] 0808 01:11:07.342792 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:07.342934 (+ 142us) service_pool.cc:227] Handling call
0808 01:11:08.181156 (+838222us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b5800003698. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adf851 00007fe349adf702 00007fe3529cfb09 00007fe35190b730 00007fe3519d179b 00007fe3519d1821 00007fe3519811b0 00007fe351980a94 00007fe34b068c04 00007fe34b068b92
0808 01:11:08.457740 (+276584us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44289280}
I20260808 01:11:08.506938 4192 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:11:08.524993 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 423593us
I20260808 01:11:08.528661 1410 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:08.531723 1446 raft_consensus.cc:1226] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:08.517982 4197 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:08.522668 4098 raft_consensus.cc:2781] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:08.537019 1410 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:08.533373 1603 rpcz_store.cc:276] 0808 01:11:06.445420 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:06.451717 (+ 6297us) service_pool.cc:227] Handling call
0808 01:11:06.452215 (+ 498us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:06.472441 (+ 20226us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:06.472474 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:06.621298 (+148824us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b2c00042c88. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34b422e11 00007fe34b432442 00007fe34ca7ac11 00007fe34ca3f58a 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772
0808 01:11:06.689798 (+ 68500us) log.cc:842] Serialized 26 byte log entry
0808 01:11:07.216746 (+526948us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:07.275635 (+ 58889us) log.cc:842] Serialized 14 byte log entry
0808 01:11:07.275950 (+ 315us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:07.276051 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:08.373759 (+1097708us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b1800471220. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349b17638 00007fe349b1ab11 00007fe349b1aac9 00007fe349b1aa59 00007fe349b1aa21 00007fe349b19d1d 00007fe352d7b204 00007fe352d7b039 00007fe349af9dc6
0808 01:11:08.520218 (+146459us) raft_consensus.cc:1671] finished
0808 01:11:08.520238 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:08.520357 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55903,"prepare.queue_time_us":56643,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":21964,"raft.run_cpu_time_us":386,"raft.run_wall_time_us":50553,"spinlock_wait_cycles":92041344,"thread_start_us":371408,"threads_started":1,"wal-append.queue_time_us":774127}
W20260808 01:11:08.541626 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 242393us
I20260808 01:11:08.543807 1410 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:08.514218 4436 raft_consensus.cc:2781] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:08.547183 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:07.594663 1516 rpcz_store.cc:276] 0808 01:11:05.933125 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:05.933849 (+ 724us) service_pool.cc:227] Handling call
0808 01:11:05.934290 (+ 441us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:05.963889 (+ 29599us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:05.963933 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 01:11:05.964544 (+ 611us) log.cc:842] Serialized 52 byte log entry
0808 01:11:06.975901 (+1011357us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:07.571952 (+596051us) log.cc:842] Serialized 14 byte log entry
0808 01:11:07.572732 (+ 780us) log.cc:842] Serialized 14 byte log entry
0808 01:11:07.576917 (+ 4185us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:07.577001 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:07.577051 (+ 50us) raft_consensus.cc:1671] finished
0808 01:11:07.577065 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:07.577198 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7862,"raft.queue_time_us":21008,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":25720,"spinlock_wait_cycles":1057664,"thread_start_us":1555489,"threads_started":2,"wal-append.queue_time_us":1230431}
I20260808 01:11:08.548600 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:08.526276 4433 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.548640 1594 raft_consensus.cc:1226] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:08.543018 4098 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:08.552932 4436 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:08.553282 4433 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:08.553361 4098 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:08.555085 1439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2090) took 1226 ms. Trace:
I20260808 01:11:08.505985 4394 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:08.555516 1445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:51830 (request call id 27) took 2494 ms. Trace:
I20260808 01:11:08.555450 1439 rpcz_store.cc:276] 0808 01:11:07.335167 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:07.335311 (+ 144us) service_pool.cc:227] Handling call
0808 01:11:08.554400 (+1219089us) spinlock_profiling.cc:243] Waited 3.81 ms on lock 0x7b5800003698. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adf851 00007fe349adf702 00007fe3529cfb09 00007fe35190b730 00007fe3519d179b 00007fe3519d1821 00007fe3519952b0 00007fe3519807f9 00007fe3519802a2 00007fe34b066684
0808 01:11:08.554592 (+ 192us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:08.554763 (+ 171us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:08.554788 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:08.554826 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:08.554870 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:08.554921 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:08.555010 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8764288}
I20260808 01:11:08.560497 1445 rpcz_store.cc:276] 0808 01:11:06.061287 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:06.062101 (+ 814us) service_pool.cc:227] Handling call
0808 01:11:08.555292 (+2493191us) spinlock_profiling.cc:243] Waited 1.96 s on lock 0x7b54001d2c54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca39bc3 00007fe35198381a 00007fe34b076d04 00007fe34b076c92 00007fe34b076bc1 00007fe34b076b3c 00007fe34b075df2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:08.555410 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4501199744}
I20260808 01:11:08.558419 4264 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:08.567373 4040 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:08.568261 4040 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:08.570490 2202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1054) took 1154 ms. Trace:
I20260808 01:11:08.572739 4165 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:08.508285 4375 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:08.574008 2201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1052) took 1171 ms. Trace:
I20260808 01:11:08.574072 4295 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:08.575446 4406 raft_consensus.cc:2781] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:08.575945 4406 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:08.576436 4406 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:08.576884 4375 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:08.572656 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 480645us
W20260808 01:11:08.581175 1487 leader_election.cc:334] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.956s (SENT)
I20260808 01:11:08.581830 1487 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:08.573082 4378 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:08.583335 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:08.567404 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:08.517869 4275 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:08.594064 1408 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:08.574563 2201 rpcz_store.cc:276] 0808 01:11:07.402511 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:07.402867 (+ 356us) service_pool.cc:227] Handling call
0808 01:11:07.403619 (+ 752us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:07.444408 (+ 40789us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:07.444462 (+ 54us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:07.445115 (+ 653us) log.cc:842] Serialized 26 byte log entry
0808 01:11:07.875368 (+430253us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:08.122562 (+247194us) log.cc:842] Serialized 14 byte log entry
0808 01:11:08.573617 (+451055us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:08.573713 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:08.573766 (+ 53us) raft_consensus.cc:1671] finished
0808 01:11:08.573782 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:08.573928 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":500352,"prepare.queue_time_us":176244,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":117212,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":9872,"thread_start_us":629664,"threads_started":2,"wal-append.queue_time_us":699429}
I20260808 01:11:08.603197 2205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1055) took 1186 ms. Trace:
I20260808 01:11:08.523881 3861 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:08.605715 1408 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:08.607065 1408 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:08.607903 4009 raft_consensus.cc:2781] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:08.608467 4009 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:08.608729 4385 raft_consensus.cc:2781] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:08.609094 4009 raft_consensus.cc:3037] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:08.609367 2171 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:08.609526 4385 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:08.610409 4385 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:08.610854 2171 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:08.612344 4420 raft_consensus.cc:2726] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:11:08.614070 4415 raft_consensus.cc:2726] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:11:08.616075 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:08.616516 2171 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:08.616804 1408 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:08.616847 2203 raft_consensus.cc:3037] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:08.616127 4424 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:08.619340 4424 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.544183 4192 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.541251 4293 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:08.553467 4436 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:08.540994 4386 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:08.671214 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:08.570976 2202 rpcz_store.cc:276] 0808 01:11:07.416063 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:07.421471 (+ 5408us) service_pool.cc:227] Handling call
0808 01:11:07.421893 (+ 422us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:07.445798 (+ 23905us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:07.445847 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:07.446380 (+ 533us) log.cc:842] Serialized 26 byte log entry
0808 01:11:08.061303 (+614923us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:08.086587 (+ 25284us) log.cc:842] Serialized 14 byte log entry
0808 01:11:08.569793 (+483206us) spinlock_profiling.cc:243] Waited 9.77 ms on lock 0x7b1800099580. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349a8e4ca 00007fe35316ff99 00007fe349952990 00007fe349b14864 00007fe349b14155 00007fe34ca373d0 00007fe34ca49576 00007fe34ca5a0d7 00007fe34ca5a058
0808 01:11:08.569991 (+ 198us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:08.570089 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:08.570148 (+ 59us) raft_consensus.cc:1671] finished
0808 01:11:08.570264 (+ 116us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:08.570417 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62559,"prepare.queue_time_us":23854,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":643638,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":16296,"spinlock_wait_cycles":22466304,"thread_start_us":611243,"threads_started":1,"wal-append.queue_time_us":699258}
W20260808 01:11:08.635160 1517 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58266 (request call id 2114) took 2873 ms (client timeout 2254 ms). Trace:
W20260808 01:11:08.672570 1517 rpcz_store.cc:269] 0808 01:11:05.761820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:05.762001 (+ 181us) service_pool.cc:227] Handling call
0808 01:11:08.634937 (+2872936us) spinlock_profiling.cc:243] Waited 2.19 s on lock 0x7b54000c7e54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:08.635077 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5046246144}
W20260808 01:11:08.645666 1565 leader_election.cc:334] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.910s (SENT)
I20260808 01:11:08.676071 1565 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:08.678136 2171 connection.cc:232] RPC call timeout handler was delayed by -1.8061678340018261s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1516s for a response
W20260808 01:11:08.678556 2171 connection.cc:232] RPC call timeout handler was delayed by -1.72966783400625s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16010000000000002s for a response
W20260808 01:11:08.678853 2171 connection.cc:232] RPC call timeout handler was delayed by -1.6531678339815699s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1686s for a response
W20260808 01:11:08.679102 2171 connection.cc:232] RPC call timeout handler was delayed by -1.6414678339788225s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16990000000000002s for a response
W20260808 01:11:08.679327 2171 connection.cc:232] RPC call timeout handler was delayed by -1.6243678339815233s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1718s for a response
W20260808 01:11:08.679550 2171 connection.cc:232] RPC call timeout handler was delayed by -1.549667833984131s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1801s for a response
W20260808 01:11:08.679801 2171 connection.cc:232] RPC call timeout handler was delayed by -1.5060621830052696s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
W20260808 01:11:08.680220 2171 connection.cc:232] RPC call timeout handler was delayed by -1.4578678339894395s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19030000000000002s for a response
W20260808 01:11:08.680685 2171 connection.cc:232] RPC call timeout handler was delayed by -1.4245678339793812s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.194s for a response
I20260808 01:11:08.636994 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:08.681882 2200 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:08.644655 1447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2096) took 1300 ms. Trace:
I20260808 01:11:08.563421 4291 raft_consensus.cc:2781] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:08.612142 4399 consensus_queue.cc:1035] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260808 01:11:08.681123 2171 connection.cc:232] RPC call timeout handler was delayed by -1.4110678340075538s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1955s for a response
I20260808 01:11:08.579551 4353 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:08.595315 4427 log.cc:924] Time spent T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.145s user 0.001s sys 0.000s
I20260808 01:11:08.619298 4040 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:11:08.699945 2171 connection.cc:232] RPC call timeout handler was delayed by -1.3183678339992184s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20579999999999998s for a response
I20260808 01:11:08.557807 4421 raft_consensus.cc:2781] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:08.603688 2205 rpcz_store.cc:276] 0808 01:11:07.416419 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:07.422301 (+ 5882us) service_pool.cc:227] Handling call
0808 01:11:07.422723 (+ 422us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:08.030223 (+607500us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:08.030269 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:08.072421 (+ 42152us) log.cc:842] Serialized 26 byte log entry
0808 01:11:08.166979 (+ 94558us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:08.167385 (+ 406us) log.cc:842] Serialized 14 byte log entry
0808 01:11:08.602813 (+435428us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:08.602914 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:08.602969 (+ 55us) raft_consensus.cc:1671] finished
0808 01:11:08.602982 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:08.603135 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":458608,"prepare.queue_time_us":427,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":637320,"raft.run_cpu_time_us":326,"raft.run_wall_time_us":29592,"thread_start_us":635189,"threads_started":2,"wal-append.queue_time_us":275701}
I20260808 01:11:08.704198 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:08.607278 3861 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:08.706969 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:08.640188 4369 consensus_meta.cc:287] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.616s user 0.009s sys 0.001s
I20260808 01:11:08.683122 1447 rpcz_store.cc:276] 0808 01:11:07.343865 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:07.418413 (+ 74548us) service_pool.cc:227] Handling call
0808 01:11:07.418714 (+ 301us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:07.418912 (+ 198us) raft_consensus.cc:1520] Early marking committed up to index 3
0808 01:11:07.419225 (+ 313us) log.cc:842] Serialized 14 byte log entry
0808 01:11:08.643810 (+1224585us) spinlock_profiling.cc:243] Waited 942 us on lock 0x7b180014a700. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349a8e4ca 00007fe35316ff99 00007fe349952990 00007fe349b14864 00007fe349b14155 00007fe350414167 00007fe34ca49807 00007fe34ca58f0f 00007fe34ca58e88
0808 01:11:08.644077 (+ 267us) log.cc:842] Serialized 14 byte log entry
0808 01:11:08.644272 (+ 195us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:08.644326 (+ 54us) raft_consensus.cc:1631] Marking committed up to 3
0808 01:11:08.644389 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:08.644469 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:08.644590 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":813866,"prepare.queue_time_us":1234876,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":75,"raft.queue_time_us":25408,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":12841,"spinlock_wait_cycles":2166656,"thread_start_us":371706,"threads_started":1,"wal-append.queue_time_us":724706}
I20260808 01:11:08.614006 4127 raft_consensus.cc:2781] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:08.764240 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:08.764976 2171 connection.cc:232] RPC call timeout handler was delayed by -1.2994678340037353s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20790000000000003s for a response
I20260808 01:11:08.712463 1594 raft_consensus.cc:1226] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 7. Current term is 8. Ops: []
W20260808 01:11:08.766757 2171 connection.cc:232] RPC call timeout handler was delayed by -1.2283678340027109s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2158s for a response
W20260808 01:11:08.767257 2171 connection.cc:232] RPC call timeout handler was delayed by -1.2130678340035956s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2175s for a response
W20260808 01:11:08.767640 2171 connection.cc:232] RPC call timeout handler was delayed by -1.1788678339798935s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22130000000000002s for a response
I20260808 01:11:08.769007 2171 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:08.714223 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:08.772914 1603 raft_consensus.cc:1261] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:11:08.776647 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:08.776846 1565 leader_election.cc:334] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.401s (SENT)
I20260808 01:11:08.623536 4413 raft_consensus.cc:2726] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:11:08.779560 2171 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:08.635334 4099 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:08.637739 4394 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:08.786835 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:08.789234 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:08.794811 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:08.701440 4289 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:08.699337 4291 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:08.714805 4295 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:08.771859 4421 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:08.807677 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:08.784015 4273 raft_consensus.cc:2726] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:08.816370 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:08.806355 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:08.821762 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:08.822459 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:08.823035 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:08.823210 1445 raft_consensus.cc:2454] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:08.821089 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1053) took 1405 ms. Trace:
I20260808 01:11:08.623629 4435 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.185s
I20260808 01:11:08.701304 4264 consensus_queue.cc:1035] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.103s
I20260808 01:11:08.600773 4378 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.800256 4127 raft_consensus.cc:686] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:08.843401 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:08.842854 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:08.889468 1601 raft_consensus.cc:2396] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260808 01:11:08.824282 2208 rpcz_store.cc:276] 0808 01:11:07.415646 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:07.420588 (+ 4942us) service_pool.cc:227] Handling call
0808 01:11:07.421084 (+ 496us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:07.956951 (+535867us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:07.956996 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 01:11:08.019766 (+ 62770us) log.cc:842] Serialized 52 byte log entry
0808 01:11:08.220852 (+201086us) raft_consensus.cc:1631] Marking committed up to 8
0808 01:11:08.221713 (+ 861us) log.cc:842] Serialized 14 byte log entry
0808 01:11:08.604307 (+382594us) log.cc:842] Serialized 14 byte log entry
0808 01:11:08.820724 (+216417us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:08.820804 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:08.820860 (+ 56us) raft_consensus.cc:1671] finished
0808 01:11:08.820885 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:08.821022 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":382064,"prepare.queue_time_us":984,"prepare.run_cpu_time_us":131,"prepare.run_wall_time_us":135,"raft.queue_time_us":575110,"raft.run_cpu_time_us":412,"raft.run_wall_time_us":20775,"thread_start_us":832403,"threads_started":3,"wal-append.queue_time_us":421730}
I20260808 01:11:08.852612 1440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 671) took 1254 ms. Trace:
I20260808 01:11:08.809417 4291 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:08.855234 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:08.826978 2200 raft_consensus.cc:2454] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
W20260808 01:11:08.859464 4443 log.cc:924] Time spent T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.258s user 0.000s sys 0.001s
I20260808 01:11:08.860255 4399 raft_consensus.cc:2726] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:08.861729 4087 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:11:08.864845 4275 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:08.865741 4293 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:08.809176 4289 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.841472 4057 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.780318 4369 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.874605 4009 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.890466 1440 rpcz_store.cc:276] 0808 01:11:07.598267 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:08.445938 (+847671us) service_pool.cc:227] Handling call
0808 01:11:08.452842 (+ 6904us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:08.453080 (+ 238us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 01:11:08.706569 (+253489us) log.cc:842] Serialized 14 byte log entry
0808 01:11:08.852051 (+145482us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:08.852135 (+ 84us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:08.852220 (+ 85us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:08.852320 (+ 100us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:08.852524 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":193314,"prepare.queue_time_us":411281,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":60,"raft.queue_time_us":162,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":131,"thread_start_us":144599,"threads_started":1}
I20260808 01:11:08.818828 4421 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:08.823781 1447 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:08.842034 4274 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:08.897430 1408 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:08.903311 1443 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a19c1759594d4722abadf576352f9a98"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51830
W20260808 01:11:08.894143 4392 consensus_meta.cc:287] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.539s user 0.008s sys 0.004s
I20260808 01:11:08.918422 4392 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.919780 4041 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:08.920500 4392 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:08.895643 4192 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:08.895589 4098 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:08.910416 1443 raft_consensus.cc:595] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:08.825726 4295 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:08.908854 4127 consensus_queue.cc:237] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.917655 1600 raft_consensus.cc:2379] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 5: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:08.896948 4413 raft_consensus.cc:2726] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:11:08.908612 4385 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.928715 4293 raft_consensus.cc:2781] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:08.943027 1597 raft_consensus.cc:1226] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 5. Current term is 6. Ops: []
W20260808 01:11:08.957228 4391 log.cc:924] Time spent T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 1.198s user 0.008s sys 0.008s
I20260808 01:11:08.952178 4399 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:08.973606 1439 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:08.974086 4399 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:08.976907 4098 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:08.978116 1446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 670) took 1442 ms. Trace:
I20260808 01:11:08.978528 1446 rpcz_store.cc:276] 0808 01:11:07.535328 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:07.764370 (+229042us) service_pool.cc:227] Handling call
0808 01:11:07.764760 (+ 390us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:08.540473 (+775713us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:08.977814 (+437341us) spinlock_profiling.cc:243] Waited 767 ms on lock 0x7b54001beed4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:08.978048 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1763369216}
I20260808 01:11:08.990092 4385 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:08.999082 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:09.000043 1444 raft_consensus.cc:2454] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:11:09.000615 1444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55040 (request call id 2091) took 1662 ms. Trace:
I20260808 01:11:09.000979 1444 rpcz_store.cc:276] 0808 01:11:07.337834 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:07.338071 (+ 237us) service_pool.cc:227] Handling call
0808 01:11:08.998868 (+1660797us) spinlock_profiling.cc:243] Waited 21.2 ms on lock 0x7b5800003698. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adf851 00007fe349adf702 00007fe3529cfb09 00007fe35190b730 00007fe3519d179b 00007fe3519d1821 00007fe3519811b0 00007fe351980a94 00007fe34b068c04 00007fe34b068b92
0808 01:11:09.000548 (+ 1680us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48742144}
W20260808 01:11:09.040797 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 528612us
W20260808 01:11:09.052865 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 256805us
W20260808 01:11:09.047055 4456 log.cc:924] Time spent T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.074s user 0.000s sys 0.005s
I20260808 01:11:09.061573 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:09.062450 2200 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:09.010558 4293 raft_consensus.cc:686] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
W20260808 01:11:09.066002 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 286528us
I20260808 01:11:09.075680 2171 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:09.083869 2171 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:09.092161 4115 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:09.092967 4115 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:09.095696 4115 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:09.098289 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:09.053839 4369 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:09.099877 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:09.100653 1596 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:09.056845 4057 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:09.057965 4295 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:09.053700 4446 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:09.106153 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:09.110222 2171 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.111428 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:09.114342 2208 raft_consensus.cc:3037] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:09.114578 2171 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:09.089138 4415 raft_consensus.cc:2781] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:09.059525 4393 consensus_queue.cc:1046] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20260808 01:11:09.126703 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:09.120101 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:09.103344 4009 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:09.134344 2200 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:09.058179 4453 consensus_queue.cc:1046] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260808 01:11:09.141000 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:09.113267 4369 raft_consensus.cc:2741] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260808 01:11:09.147823 1604 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:09.088950 4273 consensus_queue.cc:1035] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260808 01:11:09.145602 4461 raft_consensus.cc:2726] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:09.128716 4415 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:09.132694 4134 raft_consensus.cc:2781] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:09.122808 4399 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:09.105955 4289 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:09.171787 2208 raft_consensus.cc:2454] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:09.121416 4446 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:09.189601 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:09.219091 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:09.257083 4416 consensus_meta.cc:287] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.714s user 0.012s sys 0.006s
I20260808 01:11:09.277825 1438 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:09.278023 4416 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:09.280656 4416 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:09.197083 4415 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:09.215251 4369 raft_consensus.cc:2741] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260808 01:11:09.147850 4424 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:09.272851 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:09.161623 4293 consensus_queue.cc:237] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:09.211738 2203 consensus_meta.cc:287] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.586s user 0.004s sys 0.018s
I20260808 01:11:09.286628 2203 raft_consensus.cc:2454] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:09.200626 4378 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:09.294198 4446 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:09.257862 4406 consensus_meta.cc:287] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.681s user 0.000s sys 0.018s
I20260808 01:11:09.295680 4406 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:09.160790 4457 log.cc:924] Time spent T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260808 01:11:09.300554 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:09.301489 1603 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:11:09.304087 2201 raft_consensus.cc:1226] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:09.306625 1522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 1998) took 1226 ms. Trace:
W20260808 01:11:09.308156 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 768339us
I20260808 01:11:09.307024 1522 rpcz_store.cc:276] 0808 01:11:08.080150 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:08.080263 (+ 113us) service_pool.cc:227] Handling call
0808 01:11:08.111135 (+ 30872us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:08.299564 (+188429us) raft_consensus.cc:1520] Early marking committed up to index 7
0808 01:11:08.390162 (+ 90598us) log.cc:842] Serialized 14 byte log entry
0808 01:11:08.526115 (+135953us) log.cc:842] Serialized 14 byte log entry
0808 01:11:08.986384 (+460269us) spinlock_profiling.cc:243] Waited 2.29 ms on lock 0x7b1800068140. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349b17764 00007fe349b1ab11 00007fe349b1aac9 00007fe349b1aa59 00007fe349b1aa21 00007fe349b19d1d 00007fe352d7b204 00007fe352d7b039 00007fe349af9dc6
0808 01:11:09.306133 (+319749us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b1800068140. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349a8e4ca 00007fe35316ff99 00007fe349952990 00007fe349b14864 00007fe349b14155 00007fe34ca373d0 00007fe34ca49576 00007fe34ca5a0d7 00007fe34ca5a058
0808 01:11:09.306280 (+ 147us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:09.306325 (+ 45us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:09.306371 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:09.306431 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:09.306568 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2161,"prepare.queue_time_us":92692,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":70,"raft.queue_time_us":2881,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":5896,"spinlock_wait_cycles":9660416,"thread_start_us":135025,"threads_started":1,"wal-append.queue_time_us":205868}
I20260808 01:11:09.311450 4197 consensus_queue.cc:1046] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:09.313006 4113 consensus_queue.cc:1046] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:11:09.317068 4437 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:09.317936 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.318701 1517 raft_consensus.cc:2454] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 2.
I20260808 01:11:09.318627 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.319702 1524 raft_consensus.cc:2396] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
W20260808 01:11:09.319989 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 738796us
I20260808 01:11:09.321336 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.322011 1524 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:09.323060 1604 raft_consensus.cc:2454] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:09.323206 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:09.323863 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 738484us
I20260808 01:11:09.323598 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.324961 1524 raft_consensus.cc:2396] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260808 01:11:09.319285 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:09.327045 1438 raft_consensus.cc:2396] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:11:09.326982 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.328174 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.328752 1518 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:09.329674 1438 raft_consensus.cc:1261] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:11:09.329800 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.330684 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:09.331300 1523 raft_consensus.cc:3037] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 5
I20260808 01:11:09.336215 4352 raft_consensus.cc:3032] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:09.336989 4352 raft_consensus.cc:729] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:09.340229 4367 raft_consensus.cc:2726] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:09.348730 4421 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:09.349804 4113 raft_consensus.cc:3032] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:09.350463 4113 raft_consensus.cc:729] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:09.350826 1446 raft_consensus.cc:1226] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:09.352733 1442 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:11:09.348886 4387 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:09.351948 4113 consensus_queue.cc:260] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:09.353945 4387 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:09.355199 4166 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:09.356031 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.317036 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.356422 4166 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:09.357702 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.355159 4113 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:09.317025 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.362092 1522 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:09.358160 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.362948 1516 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:09.356941 1518 raft_consensus.cc:2454] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:11:09.318166 4437 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:09.358989 1524 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 10.
I20260808 01:11:09.364727 1525 raft_consensus.cc:2454] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:11:09.360239 4387 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:09.358180 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:09.371968 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:09.378424 1339 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37234 (request call id 235) took 15690 ms (client timeout 15000 ms). Trace:
I20260808 01:11:09.383915 4057 raft_consensus.cc:3032] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:09.388622 4057 raft_consensus.cc:729] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:09.390635 4057 consensus_queue.cc:260] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:09.378445 4134 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:09.396344 4387 consensus_queue.cc:1046] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:11:09.431229 1596 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:09.400246 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:09.415565 1439 raft_consensus.cc:2454] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:09.338538 4352 consensus_queue.cc:260] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:09.348839 1517 raft_consensus.cc:1261] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:11:09.285642 1601 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:11:09.427062 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 312518us
I20260808 01:11:09.437997 2171 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.367247 4437 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:09.401467 4421 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:09.439780 2171 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.444931 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.443073 2171 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.454931 2171 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:09.459893 1601 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:09.469591 2171 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.471176 2171 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.478819 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.485122 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:09.486049 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:09.489869 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.435072 4134 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:09.442873 4269 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:09.493618 2208 raft_consensus.cc:2379] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:09.494380 1568 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:09.455972 4378 raft_consensus.cc:2726] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:11:09.457054 4113 raft_consensus.cc:3032] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:09.477137 4057 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:09.480598 4289 raft_consensus.cc:2726] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:09.468076 4461 raft_consensus.cc:2726] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:09.507153 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:09.501569 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.514199 1520 raft_consensus.cc:2379] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 5: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
W20260808 01:11:09.385694 1339 rpcz_store.cc:269] 0808 01:10:53.688486 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:10:53.688796 (+ 310us) service_pool.cc:227] Handling call
0808 01:10:53.718892 (+ 30096us) spinlock_profiling.cc:243] Waited 26 ms on lock 0x7b5400002dd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4af0 00007fe34ca47f2a 00007fe352999d6c 00007fe352a93c3e 00007fe350da5fb4 00007fe350da5f42 00007fe350da5e71 00007fe350da5dec 00007fe350da50a2 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:09.378330 (+15659438us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 01:11:07.148046 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:11:07.148599 (+ 553us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:11:07.148639 (+ 40us) write_op.cc:435] Acquired schema lock
0808 01:11:07.148656 (+ 17us) tablet.cc:660] Decoding operations
0808 01:11:07.149075 (+ 419us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:11:07.149129 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0808 01:11:07.149156 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
0808 01:11:07.149514 (+ 358us) tablet.cc:699] Row locks acquired
0808 01:11:07.149529 (+ 15us) write_op.cc:260] PREPARE: finished
0808 01:11:07.149687 (+ 158us) write_op.cc:270] Start()
0808 01:11:07.149794 (+ 107us) write_op.cc:276] Timestamp: P: 1786151467149634 usec, L: 0
0808 01:11:07.149814 (+ 20us) op_driver.cc:331] REPLICATION: starting
0808 01:11:07.150540 (+ 726us) log.cc:842] Serialized 919 byte log entry
0808 01:11:08.151057 (+1000517us) op_driver.cc:444] REPLICATION: finished
0808 01:11:08.570478 (+419421us) write_op.cc:301] APPLY: starting
0808 01:11:08.570617 (+ 139us) tablet.cc:1365] starting BulkCheckPresence
0808 01:11:08.570856 (+ 239us) tablet.cc:1368] finished BulkCheckPresence
0808 01:11:08.571221 (+ 365us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 01:11:08.574605 (+ 3384us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 01:11:08.574968 (+ 363us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0808 01:11:08.575065 (+ 97us) write_op.cc:312] APPLY: finished
0808 01:11:08.575411 (+ 346us) log.cc:842] Serialized 29 byte log entry
0808 01:11:08.576715 (+ 1304us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:11:08.576861 (+ 146us) write_op.cc:454] Released schema lock
0808 01:11:08.576925 (+ 64us) write_op.cc:341] FINISH: Updating metrics
0808 01:11:08.578484 (+ 1559us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:11:08.578538 (+ 54us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7661,"reported_tablets":7,"spinlock_wait_cycles":59735424,"child_traces":[["op",{"apply.queue_time_us":419072,"apply.run_cpu_time_us":3996,"apply.run_wall_time_us":8476,"num_ops":2,"prepare.queue_time_us":677881,"prepare.run_cpu_time_us":30300,"prepare.run_wall_time_us":625247,"raft.queue_time_us":164804,"raft.run_cpu_time_us":66004,"raft.run_wall_time_us":338814,"replication_time_us":1001136,"thread_start_us":1508027,"threads_started":4,"wal-append.queue_time_us":701085}]]}
W20260808 01:11:09.495643 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 170918us
I20260808 01:11:09.515444 1490 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.483428 4399 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:09.518924 1446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2109) took 1050 ms. Trace:
I20260808 01:11:09.519452 1446 rpcz_store.cc:276] 0808 01:11:08.468800 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:08.979606 (+510806us) service_pool.cc:227] Handling call
0808 01:11:08.988644 (+ 9038us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:09.351324 (+362680us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:09.518616 (+167292us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b540007dd54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:09.518850 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":832476416}
I20260808 01:11:09.487351 4420 raft_consensus.cc:2726] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:09.526918 1442 raft_consensus.cc:1226] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:09.494933 4352 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:09.516825 1490 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.537482 1490 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.531751 1448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2103) took 1235 ms. Trace:
I20260808 01:11:09.541458 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.542287 1490 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.542739 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.497180 4406 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:09.553303 2201 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:09.555199 2207 raft_consensus.cc:2354] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5.
I20260808 01:11:09.505141 4113 raft_consensus.cc:729] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:09.510630 4098 raft_consensus.cc:2781] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:09.503394 4387 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:09.522814 4437 raft_consensus.cc:2726] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:09.576457 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.576637 1523 raft_consensus.cc:2454] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:09.576460 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:09.572645 4275 raft_consensus.cc:2726] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:09.577554 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:09.602699 1522 raft_consensus.cc:3037] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:09.578269 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.578053 4113 consensus_queue.cc:260] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:09.578886 1447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2108) took 1121 ms. Trace:
I20260808 01:11:09.601893 2201 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:09.582078 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.607867 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:09.550948 4317 raft_consensus.cc:2726] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:09.590828 1442 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:11:09.592430 2171 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:09.540840 4438 raft_consensus.cc:2781] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:09.620560 1442 raft_consensus.cc:595] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.511104 4472 raft_consensus.cc:2726] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:09.596989 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:09.623205 1446 raft_consensus.cc:2454] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:09.624342 1408 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:09.624645 2171 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.625855 2171 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.575155 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:09.600212 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:09.627992 2205 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:09.609336 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.604043 1520 raft_consensus.cc:3037] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 5
I20260808 01:11:09.584053 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.606148 1447 rpcz_store.cc:276] 0808 01:11:08.457598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:08.923275 (+465677us) service_pool.cc:227] Handling call
0808 01:11:09.037325 (+114050us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:09.037627 (+ 302us) raft_consensus.cc:1520] Early marking committed up to index 8
0808 01:11:09.054826 (+ 17199us) log.cc:842] Serialized 14 byte log entry
0808 01:11:09.578320 (+523494us) log.cc:842] Serialized 14 byte log entry
0808 01:11:09.578523 (+ 203us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:09.578566 (+ 43us) raft_consensus.cc:1631] Marking committed up to 8
0808 01:11:09.578617 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:09.578680 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:09.578818 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":429593,"prepare.queue_time_us":18762,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":79,"raft.queue_time_us":432375,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":166,"thread_start_us":93300,"threads_started":1,"wal-append.queue_time_us":468997}
I20260808 01:11:09.611182 4113 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:09.606769 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:09.575704 4098 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:09.633373 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:09.561406 4399 raft_consensus.cc:686] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:09.604748 4319 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:09.605121 2203 raft_consensus.cc:1261] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:11:09.612649 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.570452 4453 consensus_queue.cc:1225] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer 3fb1a6380d27401a831cfe5f86a4cc9d log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260808 01:11:09.613277 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:09.614138 4291 consensus_meta.cc:287] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.568s user 0.008s sys 0.004s
W20260808 01:11:09.597246 4436 consensus_meta.cc:287] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.889s user 0.017s sys 0.005s
I20260808 01:11:09.500635 4269 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:09.556921 4366 raft_consensus.cc:2726] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:09.541702 1448 rpcz_store.cc:276] 0808 01:11:08.296616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:08.502793 (+206177us) service_pool.cc:227] Handling call
0808 01:11:08.516433 (+ 13640us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:08.516655 (+ 222us) raft_consensus.cc:1520] Early marking committed up to index 4
0808 01:11:08.898001 (+381346us) log.cc:842] Serialized 14 byte log entry
0808 01:11:09.126031 (+228030us) log.cc:842] Serialized 14 byte log entry
0808 01:11:09.302726 (+176695us) spinlock_profiling.cc:243] Waited 4.6 ms on lock 0x7b1800286840. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349b17638 00007fe349b1ab11 00007fe349b1aac9 00007fe349b1aa59 00007fe349b1aa21 00007fe349b19d1d 00007fe352d7b204 00007fe352d7b039 00007fe349af9dc6
0808 01:11:09.531152 (+228426us) spinlock_profiling.cc:243] Waited 980 us on lock 0x7b1800286840. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349a8e4ca 00007fe35316ff99 00007fe349952990 00007fe349b14864 00007fe349b14155 00007fe34ca373d0 00007fe34ca49576 00007fe34ca5a0d7 00007fe34ca5a058
0808 01:11:09.531325 (+ 173us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:09.531372 (+ 47us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:09.531438 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:09.531522 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:09.531671 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":319738,"prepare.queue_time_us":349042,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":77,"raft.queue_time_us":439379,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":179,"spinlock_wait_cycles":25847808,"thread_start_us":282881,"threads_started":2,"wal-append.queue_time_us":257550}
I20260808 01:11:09.628072 2207 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:09.629643 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:09.629792 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:09.610502 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:09.632143 2171 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.626928 4416 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:09.675807 1519 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:09.605320 4387 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:09.660420 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 62054.2us
I20260808 01:11:09.641240 1522 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:09.661223 1520 raft_consensus.cc:2454] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:09.708163 1410 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:09.663312 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.708457 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:09.666308 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.708551 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.709167 1410 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:09.670636 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:09.710294 1442 raft_consensus.cc:2354] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5.
I20260808 01:11:09.631687 4317 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:09.711570 1523 raft_consensus.cc:2379] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 4: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:09.631669 4473 raft_consensus.cc:2726] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:09.672973 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:09.630545 4378 raft_consensus.cc:2781] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:11:09.716776 2207 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:09.738736 2204 raft_consensus.cc:2354] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5.
I20260808 01:11:09.657125 4098 raft_consensus.cc:3037] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:09.659719 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:09.676852 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 44785.6us
I20260808 01:11:09.692488 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.692258 4399 consensus_queue.cc:237] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:09.752808 1518 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:09.666805 4319 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:09.757246 4319 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:09.666961 4461 consensus_queue.cc:1035] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260808 01:11:09.630051 4415 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:09.765461 4352 consensus_queue.cc:1035] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:11:09.766121 4415 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:09.676326 1408 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
W20260808 01:11:09.769762 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 93796.9us
I20260808 01:11:09.770612 1408 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:09.771518 4127 raft_consensus.cc:2781] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:09.772169 4127 raft_consensus.cc:686] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:09.772306 4435 raft_consensus.cc:2781] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:09.772894 4435 raft_consensus.cc:686] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:09.773165 4127 consensus_queue.cc:237] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:09.773849 4435 consensus_queue.cc:237] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:09.774365 4134 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:09.775287 4274 consensus_queue.cc:1046] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:09.776458 4274 raft_consensus.cc:3032] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:09.776810 4274 raft_consensus.cc:729] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:09.777842 4274 consensus_queue.cc:260] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:09.704459 4467 log.cc:924] Time spent T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20260808 01:11:09.630967 4420 raft_consensus.cc:2726] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:09.668078 4291 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:09.793169 4291 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:09.705617 4416 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:09.793782 4269 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:09.706653 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260808 01:11:09.796648 4438 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:09.797403 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.798765 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:11:09.666397 4295 raft_consensus.cc:2781] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:09.664408 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:09.668249 1439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55042 (request call id 2384) took 1100 ms. Trace:
I20260808 01:11:09.669078 1444 raft_consensus.cc:1226] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:09.710469 1447 raft_consensus.cc:1261] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:11:09.714140 1410 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:09.719475 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:09.673637 1487 leader_election.cc:334] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.766s (SENT)
I20260808 01:11:09.740603 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:09.672662 4436 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:09.813791 1439 rpcz_store.cc:276] 0808 01:11:08.568314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:08.782903 (+214589us) service_pool.cc:227] Handling call
0808 01:11:09.668037 (+885134us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b540007e754. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:09.668165 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":297252736}
I20260808 01:11:09.745275 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:09.814986 1487 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:09.746877 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.815965 2207 raft_consensus.cc:2354] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5.
W20260808 01:11:09.816105 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 142498us
I20260808 01:11:09.816721 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:09.816951 1487 leader_election.cc:334] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 2.606s (SENT)
I20260808 01:11:09.817371 2207 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:09.817561 1487 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.818200 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.751812 4366 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:09.799759 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:11:09.813122 1523 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:09.742236 4378 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:09.823560 4378 raft_consensus.cc:3037] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:09.721011 4189 raft_consensus.cc:2781] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:09.824532 4189 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:09.739696 4375 raft_consensus.cc:2781] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:09.825274 4375 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:09.825654 4375 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:09.749116 1516 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:09.738083 4317 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:09.827996 4099 raft_consensus.cc:2726] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:09.744081 2171 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.748072 2201 raft_consensus.cc:2379] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 2: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:09.796903 1440 raft_consensus.cc:1226] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6. Ops: []
W20260808 01:11:09.819669 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 105651us
I20260808 01:11:09.821904 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.796662 4416 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:09.832693 1410 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:09.840754 1410 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
W20260808 01:11:09.833252 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 89321.5us
I20260808 01:11:09.842754 2171 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.843694 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.845292 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.847051 1523 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:09.839025 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.819450 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.854233 2207 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 8.
I20260808 01:11:09.855396 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.856523 2207 raft_consensus.cc:2354] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5.
I20260808 01:11:09.857620 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.858764 2207 raft_consensus.cc:2379] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 4: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:09.799618 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.871059 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:09.791695 4387 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:09.869696 1599 raft_consensus.cc:1226] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:09.871780 2207 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:09.797405 4438 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:09.853550 2201 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:09.877646 1523 raft_consensus.cc:1226] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:09.863387 2171 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.824949 4189 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:09.886607 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:09.883155 1438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2116) took 1058 ms. Trace:
I20260808 01:11:09.888873 1438 rpcz_store.cc:276] 0808 01:11:08.824535 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:09.330392 (+505857us) service_pool.cc:227] Handling call
0808 01:11:09.330729 (+ 337us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:09.330922 (+ 193us) raft_consensus.cc:1520] Early marking committed up to index 2
0808 01:11:09.402199 (+ 71277us) log.cc:842] Serialized 14 byte log entry
0808 01:11:09.882615 (+480416us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:09.882685 (+ 70us) raft_consensus.cc:1631] Marking committed up to 2
0808 01:11:09.882754 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:09.882838 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:09.883006 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35844,"prepare.queue_time_us":66936,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"thread_start_us":444069,"threads_started":1}
I20260808 01:11:09.889932 1565 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:09.839643 1487 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:09.891772 1565 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:09.895222 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:09.895237 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 55675.8us
I20260808 01:11:09.898751 4269 raft_consensus.cc:2726] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:11:09.832262 4291 raft_consensus.cc:2781] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:09.832003 4295 raft_consensus.cc:686] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:09.840277 4289 raft_consensus.cc:2726] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:09.869370 4274 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:09.844291 4386 raft_consensus.cc:2781] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:11:09.900884 2203 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:09.939700 4386 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:09.940227 4386 raft_consensus.cc:3037] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:09.944499 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:09.945549 4474 log.cc:924] Time spent T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.518s user 0.002s sys 0.002s
I20260808 01:11:09.946007 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.916319 4291 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:09.948345 1443 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2112) took 1385 ms. Trace:
I20260808 01:11:09.947692 2207 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:09.948598 4291 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:09.948834 1443 rpcz_store.cc:276] 0808 01:11:08.562595 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:09.055254 (+492659us) service_pool.cc:227] Handling call
0808 01:11:09.055685 (+ 431us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:09.090355 (+ 34670us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:09.090406 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 01:11:09.091132 (+ 726us) log.cc:842] Serialized 52 byte log entry
0808 01:11:09.427251 (+336119us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:09.427284 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:09.427346 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:09.944914 (+517568us) spinlock_profiling.cc:243] Waited 24.5 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:09.947978 (+ 3064us) raft_consensus.cc:1671] finished
0808 01:11:09.948020 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:09.948279 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":447077,"raft.queue_time_us":59983,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":7891,"spinlock_wait_cycles":56364416,"thread_start_us":279374,"threads_started":1,"wal-append.queue_time_us":310505}
I20260808 01:11:09.949512 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.924785 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.916651 4401 raft_consensus.cc:2781] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:09.953032 4401 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:09.953552 4401 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:09.954349 4134 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:09.899855 4387 raft_consensus.cc:2726] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:09.954471 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.958621 2207 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:09.958911 1446 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:09.873415 4471 raft_consensus.cc:2726] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:09.965277 4436 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:09.960886 4295 consensus_queue.cc:237] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:09.968684 1444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2110) took 1481 ms. Trace:
I20260808 01:11:09.963987 4415 raft_consensus.cc:2741] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260808 01:11:09.971691 1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2568) took 1432 ms. Trace:
I20260808 01:11:09.972157 1599 rpcz_store.cc:276] 0808 01:11:08.538968 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:08.539120 (+ 152us) service_pool.cc:227] Handling call
0808 01:11:08.539524 (+ 404us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:09.872130 (+1332606us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:09.971406 (+ 99276us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b5400304654. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:09.971591 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3058528128}
I20260808 01:11:09.969254 1444 rpcz_store.cc:276] 0808 01:11:08.487178 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:09.001933 (+514755us) service_pool.cc:227] Handling call
0808 01:11:09.668826 (+666893us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:09.821080 (+152254us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:09.968421 (+147341us) spinlock_profiling.cc:243] Waited 666 ms on lock 0x7b540007e750. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:09.968607 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1532612864}
I20260808 01:11:09.978027 4166 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:09.961198 1487 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:09.959519 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:09.981022 2207 raft_consensus.cc:2362] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5. Re-sending same reply.
I20260808 01:11:09.981971 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:09.983405 2207 raft_consensus.cc:1226] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260808 01:11:09.986416 4192 raft_consensus.cc:2726] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:09.927742 4249 consensus_queue.cc:1035] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260808 01:11:09.953060 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:09.988080 1487 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:09.988878 2204 raft_consensus.cc:2354] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5.
W20260808 01:11:09.990710 4463 log.cc:924] Time spent T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.770s user 0.001s sys 0.003s
I20260808 01:11:09.993213 4433 raft_consensus.cc:2726] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:09.994786 1487 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:09.997248 4317 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:10.000870 1487 leader_election.cc:398] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571) with higher term. Message: Invalid argument: T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5.
I20260808 01:11:09.839347 4366 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:09.868228 4417 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:11:09.845208 4420 consensus_queue.cc:1035] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.093s
I20260808 01:11:09.923117 3748 consensus_queue.cc:1035] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.015s
I20260808 01:11:09.939136 4099 raft_consensus.cc:1050] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:11:09.904778 1565 leader_election.cc:398] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Vote denied by peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571) with higher term. Message: Invalid argument: T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5.
I20260808 01:11:10.011745 1565 leader_election.cc:401] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20260808 01:11:10.013275 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 108708us
I20260808 01:11:10.001274 1487 leader_election.cc:401] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:11:10.014464 1440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2107) took 1564 ms. Trace:
I20260808 01:11:10.015046 1440 rpcz_store.cc:276] 0808 01:11:08.449861 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:08.917595 (+467734us) service_pool.cc:227] Handling call
0808 01:11:08.917845 (+ 250us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:09.831555 (+913710us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:10.014118 (+182563us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b540026cfd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:10.014386 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2021067520}
I20260808 01:11:10.004470 2205 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
W20260808 01:11:10.005540 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 73894.8us
I20260808 01:11:10.017529 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:10.017771 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:10.018327 2205 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:11:10.019469 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:10.019378 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:10.020865 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:09.971195 4098 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:10.007616 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:10.001789 1519 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:09.867784 4437 consensus_queue.cc:1035] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.131s
I20260808 01:11:10.025550 1565 leader_election.cc:398] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571) with higher term. Message: Invalid argument: T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5.
I20260808 01:11:10.029433 1408 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:10.032611 1565 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:10.007962 1568 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.009s sys 0.027s
W20260808 01:11:10.033730 1568 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.527s user 0.009s sys 0.027s
W20260808 01:11:10.034166 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 527802us
I20260808 01:11:10.034578 1565 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:10.024973 4317 raft_consensus.cc:2726] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571) with higher term. Message: Invalid argument: T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5.
I20260808 01:11:10.031163 3772 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:10.036558 4366 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:10.036971 3772 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:10.005581 4417 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:10.039666 1568 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:10.032820 4319 raft_consensus.cc:2726] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:10.030014 4057 raft_consensus.cc:2726] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571) with higher term. Message: Invalid argument: T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5.
W20260808 01:11:10.042299 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 484451us
I20260808 01:11:10.052248 1440 raft_consensus.cc:1226] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260808 01:11:10.044495 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:10.127610 1525 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.000s sys 0.017s
W20260808 01:11:10.134145 1525 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.512s user 0.000s sys 0.017s
I20260808 01:11:10.134449 2204 raft_consensus.cc:2379] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 6: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:10.082368 2172 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:10.135931 2171 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:10.136631 4461 raft_consensus.cc:2781] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:10.136858 4415 raft_consensus.cc:2781] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:10.137238 4461 raft_consensus.cc:686] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:10.137425 4415 raft_consensus.cc:686] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:10.138197 4461 consensus_queue.cc:237] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:10.138269 4415 consensus_queue.cc:237] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:10.136528 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 54269.5us
I20260808 01:11:10.040933 1568 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:10.149994 3748 raft_consensus.cc:2726] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:10.067757 1487 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:10.157508 4437 raft_consensus.cc:2726] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 01:11:10.158703 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 91123.5us
I20260808 01:11:10.159221 2171 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:10.160766 1487 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:10.165611 1487 leader_election.cc:398] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571) with higher term. Message: Invalid argument: T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5.
I20260808 01:11:10.142978 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:10.076565 4421 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:10.064533 1410 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:10.061404 1565 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:10.174958 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:10.192939 1601 raft_consensus.cc:2454] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:10.049731 4432 raft_consensus.cc:2781] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:11:10.175258 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:11:10.171875 2200 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.009s sys 0.009s
W20260808 01:11:10.226137 2200 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s user 0.010s sys 0.009s
W20260808 01:11:10.126400 2206 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.008s sys 0.009s
W20260808 01:11:10.226972 2206 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s user 0.009s sys 0.009s
W20260808 01:11:10.184602 1517 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.013s sys 0.006s
W20260808 01:11:10.227947 1517 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.674s user 0.013s sys 0.006s
I20260808 01:11:10.049602 4420 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:10.041654 4436 raft_consensus.cc:2781] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:10.190268 1596 raft_consensus.cc:1261] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260808 01:11:10.194393 1487 leader_election.cc:401] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20260808 01:11:10.230681 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 65240.1us
W20260808 01:11:10.194496 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 133208us
I20260808 01:11:10.046659 2205 raft_consensus.cc:1226] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:10.176357 4413 raft_consensus.cc:2741] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260808 01:11:10.150194 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 109333us
I20260808 01:11:10.237360 1565 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:10.194621 1567 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:10.197007 1566 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.032s sys 0.056s
W20260808 01:11:10.198462 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 134059us
W20260808 01:11:10.250917 1566 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.734s user 0.033s sys 0.056s
I20260808 01:11:10.251361 1410 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:10.179148 4471 consensus_queue.cc:1046] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260808 01:11:10.169199 4189 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:10.041364 4417 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:11:10.204319 3772 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:10.233520 4436 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:10.206223 4386 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:10.206590 4438 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:10.207389 4098 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:10.207320 4421 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:10.041508 4393 raft_consensus.cc:2726] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:11:10.209080 4113 consensus_meta.cc:287] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.545s user 0.008s sys 0.004s
I20260808 01:11:10.209777 1338 catalog_manager.cc:5495] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 3 to 4, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 4 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:11:10.210942 4378 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:10.214627 4369 raft_consensus.cc:3032] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:10.216176 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:10.220162 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:10.221413 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:10.042851 4273 raft_consensus.cc:2781] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:11:10.237920 4251 consensus_queue.cc:1046] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260808 01:11:10.220203 4057 raft_consensus.cc:2726] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:10.266438 4471 consensus_queue.cc:1225] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer 3fb1a6380d27401a831cfe5f86a4cc9d log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260808 01:11:10.255438 1410 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:10.257936 1565 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:10.236382 4319 raft_consensus.cc:2726] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571) with higher term. Message: Invalid argument: T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5.
W20260808 01:11:10.271730 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 77206.9us
I20260808 01:11:10.278352 1338 catalog_manager.cc:5495] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 4 to 5, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 5 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:11:10.199903 4293 raft_consensus.cc:2726] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:10.273689 4179 raft_consensus.cc:2781] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:10.284683 1604 raft_consensus.cc:2454] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:10.290571 1522 raft_consensus.cc:2396] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260808 01:11:10.256419 4146 raft_consensus.cc:2781] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:10.229211 4432 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:10.360648 4432 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:10.360702 4274 raft_consensus.cc:3032] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:10.361275 4274 raft_consensus.cc:729] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:10.362241 4274 consensus_queue.cc:260] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:10.368500 1440 raft_consensus.cc:1226] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 7. Current term is 8. Ops: []
I20260808 01:11:10.369861 4438 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:10.371456 4378 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:10.357867 4146 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:10.372332 4146 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:10.372133 4378 consensus_queue.cc:1046] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 01:11:10.271152 3748 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:10.284368 1597 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:10.236913 4420 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:10.289667 4273 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:10.271173 4477 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:10.301223 1565 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:10.354944 1338 catalog_manager.cc:5495] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:11:10.352962 4179 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:10.351969 4269 raft_consensus.cc:2781] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:10.394356 1338 catalog_manager.cc:5495] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: config changed from index 5 to 6, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:11:10.277976 4352 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:10.271482 4436 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:10.375813 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:11:10.393253 4375 consensus_meta.cc:287] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.567s user 0.007s sys 0.006s
W20260808 01:11:10.300560 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 45224.7us
W20260808 01:11:10.283128 4435 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.503s user 0.013s sys 0.001s
I20260808 01:11:10.278165 4480 raft_consensus.cc:2726] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:10.294719 4289 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:10.381397 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:10.283155 4406 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:10.404610 4179 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:10.350720 1567 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:10.277143 4479 consensus_queue.cc:1046] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.109s
I20260808 01:11:10.352175 4416 raft_consensus.cc:2781] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election won for term 5
W20260808 01:11:10.294975 4367 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.009s sys 0.008s
I20260808 01:11:10.283732 4369 raft_consensus.cc:729] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
W20260808 01:11:10.397953 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 96803us
W20260808 01:11:10.401746 1520 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.014s sys 0.007s
W20260808 01:11:10.449676 1520 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.629s user 0.014s sys 0.007s
W20260808 01:11:10.283318 4399 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.007s sys 0.040s
I20260808 01:11:10.290416 4412 consensus_queue.cc:1046] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.094s
I20260808 01:11:10.396809 2203 raft_consensus.cc:2379] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 3: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:10.420596 1490 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:10.392580 4273 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:11:10.455559 1565 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:10.436866 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:10.397617 3748 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:10.442271 1516 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.014s sys 0.010s
I20260808 01:11:10.352308 4099 consensus_queue.cc:237] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:10.463168 1490 leader_election.cc:398] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291) with higher term. Message: Invalid argument: T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5.
I20260808 01:11:10.378201 4401 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:10.444324 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 93732.2us
I20260808 01:11:10.407616 4098 raft_consensus.cc:2781] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:10.435899 4269 raft_consensus.cc:686] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:10.452316 2204 raft_consensus.cc:1261] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260808 01:11:10.397219 4477 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:10.457453 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:10.385836 2209 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.005s sys 0.028s
I20260808 01:11:10.437502 2172 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:10.459510 1516 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.576s user 0.015s sys 0.010s
I20260808 01:11:10.448247 4483 raft_consensus.cc:3037] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:10.465648 1567 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:10.452317 4367 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.734s user 0.011s sys 0.009s
I20260808 01:11:10.453536 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:10.437876 4352 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:10.451213 4416 raft_consensus.cc:686] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:10.436375 4375 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:10.444162 4406 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:10.441103 4289 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:10.470024 2209 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s user 0.005s sys 0.028s
I20260808 01:11:10.473809 4386 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:10.471112 4189 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:10.474745 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:10.478699 2206 raft_consensus.cc:1261] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:11:10.482815 4127 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:11:10.466166 4098 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:11:10.454147 4399 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s user 0.011s sys 0.040s
I20260808 01:11:10.466612 4393 consensus_queue.cc:1046] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260808 01:11:10.469208 4369 consensus_queue.cc:260] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:10.524160 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:10.518829 1443 raft_consensus.cc:1226] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: []
W20260808 01:11:10.617547 1524 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.004s sys 0.017s
I20260808 01:11:10.510648 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:10.605530 1410 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:10.618536 2202 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s user 0.010s sys 0.018s
I20260808 01:11:10.618633 1442 raft_consensus.cc:2362] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5. Re-sending same reply.
I20260808 01:11:10.619388 1410 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:10.619860 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:10.490257 1446 pending_rounds.cc:76] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Aborting all ops after (but not including) 2
I20260808 01:11:10.617830 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:10.471088 4481 raft_consensus.cc:2726] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 01:11:10.619041 2202 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.621s user 0.010s sys 0.018s
I20260808 01:11:10.528976 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:10.654016 1448 raft_consensus.cc:2379] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 6: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:10.623099 1446 pending_rounds.cc:97] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260808 01:11:10.677580 1446 raft_consensus.cc:2866] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:10.493873 1338 catalog_manager.cc:5495] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 1 to 2, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 2 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
W20260808 01:11:10.618177 1524 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.544s user 0.004s sys 0.017s
W20260808 01:11:10.469352 4445 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.688s user 0.014s sys 0.012s
I20260808 01:11:10.469836 4479 raft_consensus.cc:2781] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:10.664443 1490 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:10.680249 1524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36560 (request call id 589) took 1109 ms. Trace:
W20260808 01:11:10.602680 4295 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.010s sys 0.016s
I20260808 01:11:10.531492 4375 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:10.491165 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:10.534927 1597 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 8.
I20260808 01:11:10.683285 2172 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:10.691864 4399 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:10.692637 4387 raft_consensus.cc:2781] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:10.528613 4406 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:10.692837 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:10.485984 4098 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:10.498534 4421 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:10.526772 4291 consensus_meta.cc:287] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.578s user 0.007s sys 0.004s
W20260808 01:11:10.693624 4295 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.725s user 0.010s sys 0.016s
I20260808 01:11:10.524302 4189 consensus_queue.cc:1035] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.151s
I20260808 01:11:10.598204 3748 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:10.573904 4413 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:10.778137 3748 raft_consensus.cc:2726] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:10.778226 4413 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:10.524428 4420 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:10.683161 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:10.701156 1521 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.016s sys 0.002s
I20260808 01:11:10.712561 1439 raft_consensus.cc:2379] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
W20260808 01:11:10.784193 1521 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.604s user 0.016s sys 0.002s
I20260808 01:11:10.686384 2200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 635) took 1196 ms. Trace:
W20260808 01:11:10.682294 4445 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.933s user 0.014s sys 0.013s
I20260808 01:11:10.525872 4289 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:10.530249 4401 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:10.800634 4483 raft_consensus.cc:2726] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 01:11:10.617584 4165 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.679s user 0.014s sys 0.018s
I20260808 01:11:10.525468 4352 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:10.681584 4437 raft_consensus.cc:2726] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:11:10.804132 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:10.805322 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:10.806378 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:10.679057 1338 catalog_manager.cc:5495] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260808 01:11:10.811085 1517 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.007s sys 0.014s
W20260808 01:11:10.811419 1517 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s user 0.007s sys 0.014s
I20260808 01:11:10.653337 4378 raft_consensus.cc:3032] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:11:10.681871 4369 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:10.733867 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:10.740022 2171 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:10.531780 4179 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:10.742524 4438 raft_consensus.cc:979] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d: : Instructing follower 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
I20260808 01:11:10.652619 3772 raft_consensus.cc:2726] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:10.531905 4416 consensus_queue.cc:237] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:10.744360 4490 log.cc:924] Time spent T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260808 01:11:10.746551 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:10.748790 4489 log.cc:924] Time spent T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260808 01:11:10.519038 4432 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:10.750123 4477 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:10.813288 4378 raft_consensus.cc:729] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:10.599908 4146 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:10.715969 4399 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:10.755241 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:10.868954 4146 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:10.869863 1438 raft_consensus.cc:1226] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:10.756392 4480 raft_consensus.cc:3032] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:10.872031 4480 raft_consensus.cc:729] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:10.873155 4480 consensus_queue.cc:260] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:10.874958 4480 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:10.757494 1516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2018) took 1054 ms. Trace:
I20260808 01:11:10.690047 1524 rpcz_store.cc:276] 0808 01:11:09.570960 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:09.571102 (+ 142us) service_pool.cc:227] Handling call
0808 01:11:09.571590 (+ 488us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:09.571932 (+ 342us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:09.571971 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:09.577233 (+ 5262us) log.cc:842] Serialized 26 byte log entry
0808 01:11:10.007719 (+430486us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:10.679426 (+671707us) log.cc:842] Serialized 14 byte log entry
0808 01:11:10.679698 (+ 272us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:10.679796 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:10.679996 (+ 200us) raft_consensus.cc:1671] finished
0808 01:11:10.680015 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:10.680179 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":468362,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":345,"thread_start_us":1035152,"threads_started":2,"wal-append.queue_time_us":624027}
I20260808 01:11:10.876961 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:10.876080 1516 rpcz_store.cc:276] 0808 01:11:09.703051 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:09.845640 (+142589us) service_pool.cc:227] Handling call
0808 01:11:09.882010 (+ 36370us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:09.882440 (+ 430us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:09.882491 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:09.882917 (+ 426us) log.cc:842] Serialized 26 byte log entry
0808 01:11:10.469123 (+586206us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:10.469169 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:10.469238 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:10.755960 (+286722us) spinlock_profiling.cc:243] Waited 9.1 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:10.757142 (+ 1182us) raft_consensus.cc:1671] finished
0808 01:11:10.757167 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:10.757431 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1089,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":21161,"spinlock_wait_cycles":20919552,"thread_start_us":585909,"threads_started":1,"wal-append.queue_time_us":862032}
I20260808 01:11:10.681984 4461 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:10.878741 4461 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:10.524572 4269 consensus_queue.cc:237] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:10.773118 4291 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:10.881776 4471 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:10.780169 2205 raft_consensus.cc:1226] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:10.700991 4488 raft_consensus.cc:2781] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:10.715099 4481 raft_consensus.cc:2726] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:10.793605 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:10.784893 2200 rpcz_store.cc:276] 0808 01:11:09.490219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:09.491102 (+ 883us) service_pool.cc:227] Handling call
0808 01:11:09.491575 (+ 473us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:09.517405 (+ 25830us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:09.517453 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:09.529386 (+ 11933us) log.cc:842] Serialized 26 byte log entry
0808 01:11:10.226614 (+697228us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:10.622446 (+395832us) log.cc:842] Serialized 14 byte log entry
0808 01:11:10.686002 (+ 63556us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:10.686096 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:10.686152 (+ 56us) raft_consensus.cc:1671] finished
0808 01:11:10.686168 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:10.686318 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":69798,"prepare.queue_time_us":309611,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":2007,"raft.queue_time_us":98390,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":21344,"spinlock_wait_cycles":15744,"thread_start_us":999957,"threads_started":2,"wal-append.queue_time_us":785898}
I20260808 01:11:10.884758 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:10.798486 4393 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:10.852193 4295 raft_consensus.cc:1067] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Signalling peer 6b09dbefd24b4d52b385c4dff5697dc5 to start an election
W20260808 01:11:10.824234 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 84342.4us
I20260808 01:11:10.852619 4433 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:11:10.844107 1519 raft_consensus.cc:1261] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260808 01:11:10.865018 4197 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s user 0.029s sys 0.029s
I20260808 01:11:10.847260 1517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2146) took 1354 ms. Trace:
I20260808 01:11:10.896957 1517 rpcz_store.cc:276] 0808 01:11:09.493238 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:09.502154 (+ 8916us) service_pool.cc:227] Handling call
0808 01:11:09.502512 (+ 358us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:09.502751 (+ 239us) raft_consensus.cc:1520] Early marking committed up to index 10
0808 01:11:10.228362 (+725611us) log.cc:842] Serialized 14 byte log entry
0808 01:11:10.824457 (+596095us) log.cc:842] Serialized 14 byte log entry
0808 01:11:10.824765 (+ 308us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:10.824840 (+ 75us) raft_consensus.cc:1631] Marking committed up to 10
0808 01:11:10.845893 (+ 21053us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:10.846010 (+ 117us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:10.847183 (+ 1173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11224,"prepare.queue_time_us":905001,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":12536,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":23260,"spinlock_wait_cycles":73721600,"thread_start_us":1257081,"threads_started":2}
I20260808 01:11:10.810796 1338 catalog_manager.cc:5495] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:11:10.900498 1338 catalog_manager.cc:5495] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 4 to 5, leader changed from 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 5 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:10.853884 2204 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
W20260808 01:11:10.750658 4485 log.cc:924] Time spent T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260808 01:11:10.825078 4406 raft_consensus.cc:2726] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 01:11:10.886868 1518 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.001s sys 0.024s
W20260808 01:11:10.801645 4165 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.863s user 0.019s sys 0.020s
W20260808 01:11:10.774391 4415 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.010s sys 0.015s
I20260808 01:11:10.775581 4274 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:10.892369 1487 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:10.894341 2206 raft_consensus.cc:1261] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260808 01:11:10.825207 2201 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.000s sys 0.021s
I20260808 01:11:10.852332 4420 consensus_queue.cc:1046] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
W20260808 01:11:10.910108 1518 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s user 0.001s sys 0.024s
I20260808 01:11:10.925588 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 636) took 1425 ms. Trace:
W20260808 01:11:10.925941 2201 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.657s user 0.001s sys 0.024s
I20260808 01:11:10.905534 1338 catalog_manager.cc:5495] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 2 to 3, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:11:10.891034 4412 raft_consensus.cc:3032] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Stepping down as leader of term 4
W20260808 01:11:10.930334 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 475035us
I20260808 01:11:10.931288 2208 rpcz_store.cc:276] 0808 01:11:09.500032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:09.500231 (+ 199us) service_pool.cc:227] Handling call
0808 01:11:09.961969 (+461738us) spinlock_profiling.cc:243] Waited 5.06 ms on lock 0x7b5800005198. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adf851 00007fe349adf702 00007fe3529cfb09 00007fe35190b730 00007fe3519d179b 00007fe3519d1821 00007fe3519952b0 00007fe3519807f9 00007fe3519802a2 00007fe34b066684
0808 01:11:09.962242 (+ 273us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:09.962731 (+ 489us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:09.962775 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:09.963401 (+ 626us) log.cc:842] Serialized 26 byte log entry
0808 01:11:10.395277 (+431876us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:10.913082 (+517805us) log.cc:842] Serialized 14 byte log entry
0808 01:11:10.913378 (+ 296us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:10.913465 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:10.913512 (+ 47us) raft_consensus.cc:1671] finished
0808 01:11:10.925355 (+ 11843us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:10.925522 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72334,"prepare.queue_time_us":202494,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":1768,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"spinlock_wait_cycles":11645312,"thread_start_us":910180,"threads_started":2,"wal-append.queue_time_us":792635}
I20260808 01:11:10.896381 4413 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:10.942698 1338 catalog_manager.cc:5495] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:10.949272 1338 catalog_manager.cc:5495] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:11:10.939529 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:10.939780 1518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2017) took 1237 ms. Trace:
I20260808 01:11:10.945363 2204 raft_consensus.cc:1261] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260808 01:11:10.942968 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 471405us
I20260808 01:11:10.911456 4488 raft_consensus.cc:686] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:10.948158 2208 raft_consensus.cc:1261] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260808 01:11:10.931488 4415 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s user 0.010s sys 0.015s
I20260808 01:11:10.954123 1338 catalog_manager.cc:5495] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 2 to 3, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:11:10.886237 4471 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:10.940817 4378 consensus_queue.cc:260] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:10.970131 4508 log.cc:924] Time spent T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260808 01:11:10.970388 4432 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:10.971920 4461 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:10.972795 4179 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:10.996160 4421 consensus_queue.cc:1035] T 821d5636d01e4067bac2bbd7475921ef P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.007s
W20260808 01:11:10.975389 4417 consensus_meta.cc:287] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.701s user 0.003s sys 0.011s
W20260808 01:11:10.980417 4510 log.cc:924] Time spent T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260808 01:11:10.980825 4291 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:10.981037 4495 log.cc:924] Time spent T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.238s user 0.001s sys 0.001s
I20260808 01:11:10.982432 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:11:10.983513 4514 log.cc:924] Time spent T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20260808 01:11:10.965965 4413 raft_consensus.cc:3032] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:10.945030 4477 raft_consensus.cc:3037] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:10.983568 4438 consensus_queue.cc:1035] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:11:10.966955 1338 catalog_manager.cc:5495] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 2 to 3, leader changed from 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 3 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:10.987411 1448 raft_consensus.cc:1261] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:11:10.895462 4393 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:10.991508 4488 consensus_queue.cc:237] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:10.936767 4412 raft_consensus.cc:729] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:11.050490 4420 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:10.939163 4386 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:10.993085 1519 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a19c1759594d4722abadf576352f9a98"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:58266
I20260808 01:11:11.086740 1519 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:11:11.087296 1519 raft_consensus.cc:3037] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:11.004408 2208 raft_consensus.cc:1261] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:11:10.947433 4479 raft_consensus.cc:686] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
W20260808 01:11:11.010056 1568 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.012s sys 0.016s
W20260808 01:11:11.088037 1568 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.742s user 0.013s sys 0.017s
W20260808 01:11:11.088436 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 742613us
I20260808 01:11:11.004807 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:10.990274 4399 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:10.961846 4367 consensus_queue.cc:1046] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260808 01:11:11.013283 4417 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:11.054685 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:11.064329 1338 catalog_manager.cc:5495] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 4 to 5, leader changed from 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 5 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:11.074079 2205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 847) took 1228 ms. Trace:
I20260808 01:11:11.109477 2205 rpcz_store.cc:276] 0808 01:11:09.845988 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:10.236233 (+390245us) service_pool.cc:227] Handling call
0808 01:11:10.779991 (+543758us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:10.884910 (+104919us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:11.073776 (+188866us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b54004538d0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:11.073997 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1196536064}
I20260808 01:11:11.001354 4189 consensus_queue.cc:1035] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.055s
I20260808 01:11:11.079785 1444 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 678) took 1036 ms. Trace:
I20260808 01:11:11.051481 4453 raft_consensus.cc:2726] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:11.055538 4317 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:11.051555 4273 consensus_meta.cc:287] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.585s user 0.014s sys 0.005s
I20260808 01:11:11.053517 4461 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:11.074950 4415 consensus_queue.cc:1035] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260808 01:11:11.052549 4413 raft_consensus.cc:729] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:11.057072 4291 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:11.100977 1525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36560 (request call id 588) took 1546 ms. Trace:
I20260808 01:11:11.135414 1525 rpcz_store.cc:276] 0808 01:11:09.554789 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:09.555608 (+ 819us) service_pool.cc:227] Handling call
0808 01:11:09.601303 (+ 45695us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:10.134624 (+533321us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:10.134667 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:10.197235 (+ 62568us) log.cc:842] Serialized 26 byte log entry
0808 01:11:10.689076 (+491841us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:10.689129 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:10.689305 (+ 176us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:11.100663 (+411358us) raft_consensus.cc:1671] finished
0808 01:11:11.100695 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:11.100897 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":71837,"raft.queue_time_us":28962,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":24159,"thread_start_us":1003648,"threads_started":2,"wal-append.queue_time_us":690190}
I20260808 01:11:11.109047 2204 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:11.110210 1338 catalog_manager.cc:5495] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } }
I20260808 01:11:11.114015 1410 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:11.139053 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:11.043689 2202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 637) took 1543 ms. Trace:
I20260808 01:11:11.117522 1444 rpcz_store.cc:276] 0808 01:11:10.043184 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:10.043383 (+ 199us) service_pool.cc:227] Handling call
0808 01:11:10.066004 (+ 22621us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:10.074305 (+ 8301us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:10.074355 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:10.080076 (+ 5721us) log.cc:842] Serialized 26 byte log entry
0808 01:11:10.571127 (+491051us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:10.571183 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:10.571286 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:10.969595 (+398309us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:11.079425 (+109830us) raft_consensus.cc:1671] finished
0808 01:11:11.079457 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:11.079724 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7777,"raft.queue_time_us":8442,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":2877,"spinlock_wait_cycles":26077952,"thread_start_us":456686,"threads_started":1,"wal-append.queue_time_us":755827}
W20260808 01:11:10.994205 2207 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.010s sys 0.013s
W20260808 01:11:11.129670 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 106596us
I20260808 01:11:11.111109 4134 raft_consensus.cc:2726] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:11.148420 2209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 638) took 1646 ms. Trace:
I20260808 01:11:11.148710 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:11.146423 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:11.148986 2209 rpcz_store.cc:276] 0808 01:11:09.501963 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:09.502135 (+ 172us) service_pool.cc:227] Handling call
0808 01:11:09.502556 (+ 421us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:09.545848 (+ 43292us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:09.545898 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:09.555051 (+ 9153us) log.cc:842] Serialized 26 byte log entry
0808 01:11:09.792760 (+237709us) spinlock_profiling.cc:243] Waited 2.93 ms on lock 0x7b1800224980. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349af9345 00007fe352930a39 00007fe349b168bb 00007fe349b14cbf 00007fe349b16e22 00007fe34c6c05bb 00007fe34c6c6063 00007fe34c6c6411 00007fe34ca135cf
0808 01:11:10.475999 (+683239us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:10.476868 (+ 869us) log.cc:842] Serialized 14 byte log entry
0808 01:11:10.477278 (+ 410us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:10.778182 (+300904us) spinlock_profiling.cc:243] Waited 33 ms on lock 0x7b54007378d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:10.778234 (+ 52us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:10.980380 (+202146us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:11.148196 (+167816us) raft_consensus.cc:1671] finished
0808 01:11:11.148219 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:11.148355 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17246,"prepare.queue_time_us":304019,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":38325,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":9192,"spinlock_wait_cycles":116235520,"thread_start_us":665324,"threads_started":1,"wal-append.queue_time_us":1150663}
I20260808 01:11:11.142340 1338 catalog_manager.cc:5495] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 2 to 3, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 3 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
W20260808 01:11:11.144090 2207 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s user 0.010s sys 0.013s
I20260808 01:11:11.144155 2202 rpcz_store.cc:276] 0808 01:11:09.500650 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:09.500806 (+ 156us) service_pool.cc:227] Handling call
0808 01:11:09.501260 (+ 454us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:09.880242 (+378982us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:09.880319 (+ 77us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:09.899814 (+ 19495us) log.cc:842] Serialized 26 byte log entry
0808 01:11:10.691663 (+791849us) raft_consensus.cc:1631] Marking committed up to 8
0808 01:11:10.773184 (+ 81521us) log.cc:842] Serialized 14 byte log entry
0808 01:11:11.040806 (+267622us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:11.040940 (+ 134us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:11.041012 (+ 72us) raft_consensus.cc:1671] finished
0808 01:11:11.041033 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:11.043591 (+ 2558us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27183,"prepare.queue_time_us":91898,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":76663,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":12872,"thread_start_us":1287066,"threads_started":3,"wal-append.queue_time_us":1051457}
I20260808 01:11:11.160629 1338 catalog_manager.cc:5495] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } } }
I20260808 01:11:11.144011 1601 raft_consensus.cc:1226] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:11.168274 1338 catalog_manager.cc:5495] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:11:11.124897 4483 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:11.095348 4420 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:11.174775 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:11.200670 1601 raft_consensus.cc:2354] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7.
I20260808 01:11:11.145243 1519 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:11.097821 4386 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:11.178615 2207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 842) took 1366 ms. Trace:
I20260808 01:11:11.177438 1338 catalog_manager.cc:5495] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 4 to 6, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 6 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } }
I20260808 01:11:11.207089 1338 catalog_manager.cc:5495] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:11.179705 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:11.136612 4387 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:11.204319 2207 rpcz_store.cc:276] 0808 01:11:09.812130 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:10.004156 (+192026us) service_pool.cc:227] Handling call
0808 01:11:10.004507 (+ 351us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:10.004719 (+ 212us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 01:11:10.486115 (+481396us) log.cc:842] Serialized 14 byte log entry
0808 01:11:11.178145 (+692030us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:11.178198 (+ 53us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:11.178251 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:11.178329 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:11.178551 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18733,"prepare.queue_time_us":466245,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":39036,"thread_start_us":1083269,"threads_started":2,"wal-append.queue_time_us":692088}
I20260808 01:11:11.214025 1520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2012) took 1544 ms. Trace:
I20260808 01:11:11.125795 4295 consensus_queue.cc:1046] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260808 01:11:11.212324 1338 catalog_manager.cc:5495] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 2 to 4, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 4 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:11.218729 1519 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:11.218052 1338 catalog_manager.cc:5495] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 reported cstate change: term changed from 5 to 6, leader changed from 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 6 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:11.221088 1601 raft_consensus.cc:1226] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:11.214383 1520 rpcz_store.cc:276] 0808 01:11:09.669239 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:09.786981 (+117742us) service_pool.cc:227] Handling call
0808 01:11:09.787341 (+ 360us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:09.787601 (+ 260us) raft_consensus.cc:1520] Early marking committed up to index 8
0808 01:11:10.450285 (+662684us) log.cc:842] Serialized 14 byte log entry
0808 01:11:10.940546 (+490261us) log.cc:842] Serialized 14 byte log entry
0808 01:11:11.213716 (+273170us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:11.213766 (+ 50us) raft_consensus.cc:1631] Marking committed up to 8
0808 01:11:11.213819 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:11.213890 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:11.213976 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24857,"prepare.queue_time_us":663939,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":71,"raft.queue_time_us":27990,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":14441,"spinlock_wait_cycles":129898112,"thread_start_us":1369993,"threads_started":3,"wal-append.queue_time_us":757702}
I20260808 01:11:11.211648 1521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2024) took 1080 ms. Trace:
I20260808 01:11:11.181743 4446 raft_consensus.cc:2726] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:11.173071 4099 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:11.129894 4273 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:11.227516 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:11.130651 4461 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:11.225015 1521 rpcz_store.cc:276] 0808 01:11:10.130675 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:10.131093 (+ 418us) service_pool.cc:227] Handling call
0808 01:11:10.131358 (+ 265us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:10.131559 (+ 201us) raft_consensus.cc:1520] Early marking committed up to index 2
0808 01:11:10.785594 (+654035us) log.cc:842] Serialized 14 byte log entry
0808 01:11:11.210065 (+424471us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:11.210142 (+ 77us) raft_consensus.cc:1631] Marking committed up to 2
0808 01:11:11.210217 (+ 75us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:11.211159 (+ 942us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:11.211316 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":344813,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":37,"raft.queue_time_us":7571,"raft.run_wall_time_us":9578,"thread_start_us":1028677,"threads_started":2}
W20260808 01:11:11.243500 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 185675us
I20260808 01:11:11.130873 4317 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:11.217847 4387 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:11.247030 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:10.956831 1518 rpcz_store.cc:276] 0808 01:11:09.702765 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:09.813336 (+110571us) service_pool.cc:227] Handling call
0808 01:11:09.813595 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:09.813825 (+ 230us) raft_consensus.cc:1520] Early marking committed up to index 3
0808 01:11:10.298687 (+484862us) log.cc:842] Serialized 14 byte log entry
0808 01:11:10.935506 (+636819us) log.cc:842] Serialized 14 byte log entry
0808 01:11:10.939050 (+ 3544us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:10.939129 (+ 79us) raft_consensus.cc:1631] Marking committed up to 3
0808 01:11:10.939234 (+ 105us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:10.939452 (+ 218us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:10.939661 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6791,"prepare.queue_time_us":652866,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":7359,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":4379,"thread_start_us":1053498,"threads_started":2}
I20260808 01:11:11.213498 4521 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:11.141103 4098 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:11.262881 4496 raft_consensus.cc:3032] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:11.135268 4412 consensus_queue.cc:260] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:11.263720 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 53607.6us
I20260808 01:11:11.207686 4357 consensus_queue.cc:1035] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.040s
I20260808 01:11:11.281296 1604 raft_consensus.cc:1226] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:11.283277 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:11.286896 1442 raft_consensus.cc:1226] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:11.142206 4291 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:11.245733 4471 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:11.205885 4483 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:11.271693 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:11.177328 4269 consensus_queue.cc:1046] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.088s
W20260808 01:11:11.274173 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 67021.6us
I20260808 01:11:11.274613 4393 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:11.278967 4417 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 11 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:11.279786 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:11.299283 1521 raft_consensus.cc:1226] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:11.271404 1567 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:11.299366 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:11.309305 1442 raft_consensus.cc:1226] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:11.309298 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:11.310434 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:11.310724 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:11.311183 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:11.311631 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:11.312686 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:11.313434 1601 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:11.317724 1602 raft_consensus.cc:1226] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 7. Current term is 8. Ops: []
I20260808 01:11:11.324566 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:11.324769 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:11.272748 4496 raft_consensus.cc:729] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:11.326120 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:11.329661 1448 raft_consensus.cc:1226] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:11.216521 4413 consensus_queue.cc:260] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:11.325258 1602 raft_consensus.cc:2454] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
W20260808 01:11:11.309441 4518 log.cc:924] Time spent T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.227s user 0.000s sys 0.001s
I20260808 01:11:11.243314 4099 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:11.343649 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 64579.7us
I20260808 01:11:11.352756 1598 raft_consensus.cc:1226] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260808 01:11:11.288050 4189 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:11.356527 1448 raft_consensus.cc:1226] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 7. Current term is 8. Ops: []
I20260808 01:11:11.357715 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:11.358420 1445 raft_consensus.cc:3032] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:11:11.358745 1445 raft_consensus.cc:729] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:11.359539 1445 consensus_queue.cc:260] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:11.360916 1445 raft_consensus.cc:3037] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:11.229017 4479 consensus_queue.cc:237] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:11.373996 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:11.374714 1604 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
W20260808 01:11:11.372967 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 86780.9us
I20260808 01:11:11.375867 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:11.376945 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:11.377812 1597 raft_consensus.cc:2454] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:11.377108 2171 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:11.378054 1489 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:11.370882 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:11.384205 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 113460us
I20260808 01:11:11.385193 1598 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 8.
I20260808 01:11:11.386716 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:11.389837 1487 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:11.393684 1601 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:11.395011 1600 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:11.396312 2203 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 854) took 1323 ms. Trace:
I20260808 01:11:11.397119 2171 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:11.353682 4412 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:11.354691 4471 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:11.345055 4534 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:11.390419 4516 log.cc:924] Time spent T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.270s user 0.000s sys 0.001s
I20260808 01:11:11.370237 4386 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:11.408563 2171 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:11.269447 4521 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:11.413717 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:11.417308 1448 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:11.448472 2171 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:11.436477 1445 raft_consensus.cc:2454] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:11.465720 2171 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:11.466583 2171 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
W20260808 01:11:11.410633 4527 log.cc:924] Time spent T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.290s user 0.001s sys 0.000s
I20260808 01:11:11.403194 4539 raft_consensus.cc:3032] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:11.478996 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:11.479980 1445 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:11.407399 4269 raft_consensus.cc:2726] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:11:11.406471 4550 raft_consensus.cc:2726] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:11.383476 4189 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:11.420811 4481 raft_consensus.cc:2726] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:11.473026 4483 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:11.490511 1439 raft_consensus.cc:1226] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:11.474306 4289 raft_consensus.cc:2781] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:11.497893 1448 raft_consensus.cc:1226] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:11.474152 4420 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:11.409502 4471 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:11.476755 4317 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:11.479493 4539 raft_consensus.cc:729] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
W20260808 01:11:11.487054 4523 log.cc:924] Time spent T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.260s user 0.000s sys 0.001s
I20260808 01:11:11.516786 1438 raft_consensus.cc:3037] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:11.446491 4435 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:11.406935 4537 consensus_queue.cc:1046] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.106s
I20260808 01:11:11.526104 2204 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:11:11.531497 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:11.498180 4273 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:11.537487 1408 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:11.501680 4415 raft_consensus.cc:2781] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:11.543650 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:11.544351 1446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55042 (request call id 2397) took 1059 ms. Trace:
I20260808 01:11:11.505226 4289 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:11.599080 1568 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:11.602237 1448 raft_consensus.cc:1226] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 11 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 10. Current term is 11. Ops: []
W20260808 01:11:11.522493 4436 consensus_meta.cc:287] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.082s user 0.010s sys 0.013s
I20260808 01:11:11.572548 1446 rpcz_store.cc:276] 0808 01:11:10.489467 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:10.489638 (+ 171us) service_pool.cc:227] Handling call
0808 01:11:10.490047 (+ 409us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:10.678278 (+188231us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:10.678334 (+ 56us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:10.678844 (+ 510us) log.cc:842] Serialized 26 byte log entry
0808 01:11:10.906726 (+227882us) raft_consensus.cc:1631] Marking committed up to 2
0808 01:11:10.906758 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:10.906824 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:11.544079 (+637255us) raft_consensus.cc:1671] finished
0808 01:11:11.544107 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:11.544280 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":184477,"threads_started":1,"wal-append.queue_time_us":377146}
I20260808 01:11:11.581286 4557 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:11.585310 4541 log.cc:924] Time spent T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.118s user 0.001s sys 0.001s
I20260808 01:11:11.412479 4496 consensus_queue.cc:260] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:11.585784 4537 raft_consensus.cc:2726] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:11:11.574419 4538 log.cc:924] Time spent T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.318s user 0.001s sys 0.000s
I20260808 01:11:11.587114 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:11.606407 1438 raft_consensus.cc:2454] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
W20260808 01:11:11.610764 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 233437us
I20260808 01:11:11.527029 4098 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:11.594695 4550 raft_consensus.cc:2781] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:11.520871 4481 raft_consensus.cc:2781] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:11:11.414291 4534 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:11.651305 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
W20260808 01:11:11.572751 4520 log.cc:924] Time spent T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.313s user 0.000s sys 0.001s
I20260808 01:11:11.528102 4461 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:11.588013 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 57860.1us
W20260808 01:11:11.657348 4524 log.cc:924] Time spent T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.362s user 0.001s sys 0.000s
I20260808 01:11:11.646574 4540 raft_consensus.cc:2726] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:11.661357 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:11.593022 4317 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:11.634249 4289 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:11:11.689754 4529 log.cc:924] Time spent T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.390s user 0.001s sys 0.001s
W20260808 01:11:11.673451 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 340480us
I20260808 01:11:11.640307 1568 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:11.643652 3772 raft_consensus.cc:2781] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:11:11.693572 2204 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:11.668522 4545 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:11.700620 4561 log.cc:924] Time spent T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20260808 01:11:11.707558 4480 consensus_meta.cc:287] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.832s user 0.005s sys 0.007s
I20260808 01:11:11.708400 4189 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:11.710373 4471 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:11.711580 4549 log.cc:924] Time spent T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.168s user 0.001s sys 0.000s
I20260808 01:11:11.716480 4319 raft_consensus.cc:3032] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:11.719467 4057 raft_consensus.cc:1050] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:11:11.722718 4421 raft_consensus.cc:3032] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:11:11.660948 1489 leader_election.cc:398] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7.
I20260808 01:11:11.670512 4291 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:11.574059 4415 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:11.751444 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:11.797789 1446 raft_consensus.cc:2379] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 6: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
W20260808 01:11:11.798188 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 408566us
W20260808 01:11:11.730453 4535 log.cc:924] Time spent T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.361s user 0.000s sys 0.001s
W20260808 01:11:11.754180 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 113997us
I20260808 01:11:11.808230 1568 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:11.396763 2203 rpcz_store.cc:276] 0808 01:11:10.072955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:10.779118 (+706163us) service_pool.cc:227] Handling call
0808 01:11:10.779501 (+ 383us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:10.779742 (+ 241us) raft_consensus.cc:1520] Early marking committed up to index 9
0808 01:11:10.823475 (+ 43733us) log.cc:842] Serialized 14 byte log entry
0808 01:11:11.279150 (+455675us) spinlock_profiling.cc:243] Waited 1.09 ms on lock 0x7b1800379240. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349af9345 00007fe352930a39 00007fe349b168bb 00007fe349b14cbf 00007fe349b16e22 00007fe34c6c05bb 00007fe34c6c6063 00007fe34c6c657b 00007fe34ca4988a
0808 01:11:11.395859 (+116709us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:11.395930 (+ 71us) raft_consensus.cc:1631] Marking committed up to 9
0808 01:11:11.395997 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:11.396097 (+ 100us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:11.396243 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45284,"prepare.queue_time_us":32017,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":26636,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"spinlock_wait_cycles":2498304,"thread_start_us":75198,"threads_started":1}
I20260808 01:11:11.543164 4435 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:11.731706 4496 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:11.653151 4273 consensus_queue.cc:1046] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.093s
I20260808 01:11:11.765993 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:11.659488 4557 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:11.688477 4481 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:11.737468 4099 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:11.765820 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:11.795028 4421 raft_consensus.cc:729] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:11.776527 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 861) took 1233 ms. Trace:
I20260808 01:11:11.846357 2206 rpcz_store.cc:276] 0808 01:11:10.543539 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:10.971363 (+427824us) service_pool.cc:227] Handling call
0808 01:11:10.971715 (+ 352us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:10.972179 (+ 464us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:10.972205 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 01:11:10.972834 (+ 629us) log.cc:842] Serialized 52 byte log entry
0808 01:11:11.113816 (+140982us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:11.113870 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:11.113965 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:11.776279 (+662314us) raft_consensus.cc:1671] finished
0808 01:11:11.776296 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:11.776453 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8089,"raft.queue_time_us":18582,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":6957,"thread_start_us":140553,"threads_started":1,"wal-append.queue_time_us":173675}
I20260808 01:11:11.775679 1594 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:11.695039 4488 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:11.782577 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:11.647857 4436 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:11.785422 4547 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:11.785925 4555 consensus_queue.cc:1046] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.063s
I20260808 01:11:11.768697 4540 raft_consensus.cc:2726] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:11.795714 1489 leader_election.cc:401] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:11:11.656551 4565 raft_consensus.cc:2781] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260808 01:11:11.859412 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 200261us
W20260808 01:11:11.859550 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 473952us
I20260808 01:11:11.860176 1489 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:11.860569 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260808 01:11:11.815349 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 318353us
I20260808 01:11:11.779067 3772 raft_consensus.cc:686] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:11.831354 2207 raft_consensus.cc:3037] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:11.834196 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:11.871349 1521 raft_consensus.cc:2396] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260808 01:11:11.835280 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:11.835490 4544 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:11.788960 4521 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:11.755589 4317 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:11.790537 4319 raft_consensus.cc:729] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:11.733366 4537 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:11.806581 4415 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:11.851140 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:11.851929 4558 consensus_queue.cc:1046] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260808 01:11:11.856002 2202 raft_consensus.cc:2379] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 4: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:11.837208 4352 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:11.838162 4098 raft_consensus.cc:2741] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260808 01:11:11.899242 1447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55042 (request call id 2401) took 1408 ms. Trace:
I20260808 01:11:11.845026 4481 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:11.911791 1523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2159) took 1482 ms. Trace:
I20260808 01:11:11.913275 1599 raft_consensus.cc:1226] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 5. Current term is 6. Ops: []
W20260808 01:11:11.914144 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 447565us
I20260808 01:11:11.914888 1447 rpcz_store.cc:276] 0808 01:11:10.491059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:10.491324 (+ 265us) service_pool.cc:227] Handling call
0808 01:11:10.491674 (+ 350us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:10.492125 (+ 451us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:10.492163 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:10.514626 (+ 22463us) log.cc:842] Serialized 26 byte log entry
0808 01:11:10.753369 (+238743us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b18004701a0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349af9345 00007fe352930a39 00007fe349b168bb 00007fe349b14cbf 00007fe349b16e22 00007fe34c6c05bb 00007fe34c6c6063 00007fe34c6c6411 00007fe34ca135cf
0808 01:11:11.107886 (+354517us) raft_consensus.cc:1631] Marking committed up to 2
0808 01:11:11.107931 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:11.108021 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:11.856900 (+748879us) raft_consensus.cc:1671] finished
0808 01:11:11.856920 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:11.899166 (+ 42246us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41414,"raft.queue_time_us":139,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":20162,"spinlock_wait_cycles":23908096,"thread_start_us":354272,"threads_started":1,"wal-append.queue_time_us":714612}
I20260808 01:11:11.776160 4545 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:11.932292 2171 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:11.893700 4565 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:11.858982 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:11.919193 1523 rpcz_store.cc:276] 0808 01:11:10.429232 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:10.478875 (+ 49643us) service_pool.cc:227] Handling call
0808 01:11:10.479271 (+ 396us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:10.517265 (+ 37994us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:10.517311 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:10.518143 (+ 832us) log.cc:842] Serialized 26 byte log entry
0808 01:11:10.998289 (+480146us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:11.050564 (+ 52275us) log.cc:842] Serialized 14 byte log entry
0808 01:11:11.051133 (+ 569us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:11.051215 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:11.724036 (+672821us) spinlock_profiling.cc:243] Waited 4.29 ms on lock 0x7b1800362620. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349b17638 00007fe349b1ab11 00007fe349b1aac9 00007fe349b1aa59 00007fe349b1aa21 00007fe349b19d1d 00007fe352d7b204 00007fe352d7b039 00007fe349af9dc6
0808 01:11:11.893324 (+169288us) raft_consensus.cc:1671] finished
0808 01:11:11.911537 (+ 18213us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:11.911712 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17484,"prepare.queue_time_us":52363,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":74,"raft.queue_time_us":19097,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":12409,"spinlock_wait_cycles":9874304,"thread_start_us":407181,"threads_started":1,"wal-append.queue_time_us":640411}
I20260808 01:11:11.843051 4480 consensus_queue.cc:1046] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260808 01:11:11.919962 3748 raft_consensus.cc:3032] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Stepping down as leader of term 2
W20260808 01:11:11.958397 2209 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.002s sys 0.009s
W20260808 01:11:11.959318 2209 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.546s user 0.003s sys 0.009s
I20260808 01:11:11.955709 4574 raft_consensus.cc:2726] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:11.936360 4099 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 7
I20260808 01:11:11.910674 4273 consensus_queue.cc:1046] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260808 01:11:11.778863 4539 consensus_queue.cc:260] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:11.900683 4483 consensus_queue.cc:1046] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260808 01:11:11.858582 4488 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:11.966747 4532 log.cc:924] Time spent T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.437s user 0.000s sys 0.002s
I20260808 01:11:11.972506 2204 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:11.943101 4567 raft_consensus.cc:2781] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:11.945715 4565 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:12.000818 4273 raft_consensus.cc:3032] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:11:11.919025 4547 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.032680 4273 raft_consensus.cc:729] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:12.033310 4435 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:12.035621 2201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 654) took 2525 ms. Trace:
I20260808 01:11:11.998339 4534 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:12.001770 4480 raft_consensus.cc:3032] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:11:11.994192 3772 consensus_queue.cc:237] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:11.918520 4352 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.005357 2207 raft_consensus.cc:2454] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:11.958243 3748 raft_consensus.cc:729] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
W20260808 01:11:12.007021 4569 log.cc:924] Time spent T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260808 01:11:11.918710 4544 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.013856 4436 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:12.015640 4557 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:12.017042 4317 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:12.019886 4550 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:12.020157 4539 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:11.911860 4537 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:12.021374 4545 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:12.022521 4412 consensus_meta.cc:287] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.611s user 0.012s sys 0.000s
I20260808 01:11:11.938359 4421 consensus_queue.cc:260] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.023416 2200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1077) took 1786 ms. Trace:
I20260808 01:11:12.025341 4057 consensus_queue.cc:237] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.028913 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 864) took 1180 ms. Trace:
I20260808 01:11:12.132671 2208 rpcz_store.cc:276] 0808 01:11:10.848176 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:10.975695 (+127519us) service_pool.cc:227] Handling call
0808 01:11:10.984094 (+ 8399us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:11.087800 (+103706us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:12.028607 (+940807us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b54004dc454. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:12.028839 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":45824896}
I20260808 01:11:12.027522 4433 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:11:12.064968 4548 log.cc:924] Time spent T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.522s user 0.004s sys 0.000s
I20260808 01:11:12.080507 2207 raft_consensus.cc:1226] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:12.036139 2201 rpcz_store.cc:276] 0808 01:11:09.509899 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:09.896585 (+386686us) service_pool.cc:227] Handling call
0808 01:11:09.896931 (+ 346us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:09.959766 (+ 62835us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 01:11:10.251535 (+291769us) log.cc:842] Serialized 14 byte log entry
0808 01:11:11.132707 (+881172us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:11.132763 (+ 56us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:11.132809 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:12.035323 (+902514us) spinlock_profiling.cc:243] Waited 62.6 ms on lock 0x7b540067d254. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:12.035379 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:12.035547 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4250,"prepare.queue_time_us":493824,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":34239,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"spinlock_wait_cycles":143978752,"thread_start_us":1149953,"threads_started":3,"wal-append.queue_time_us":739709,"wal-append.run_cpu_time_us":1032,"wal-append.run_wall_time_us":1004596}
I20260808 01:11:12.067363 4567 raft_consensus.cc:686] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:12.073123 4547 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:12.068451 4480 raft_consensus.cc:729] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:12.132383 2200 rpcz_store.cc:276] 0808 01:11:10.237209 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:10.885629 (+648420us) service_pool.cc:227] Handling call
0808 01:11:10.885971 (+ 342us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:10.886342 (+ 371us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 01:11:10.891668 (+ 5326us) log.cc:842] Serialized 14 byte log entry
0808 01:11:11.177927 (+286259us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:11.178438 (+ 511us) log.cc:842] Serialized 26 byte log entry
0808 01:11:11.178569 (+ 131us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:11.204522 (+ 25953us) log.cc:842] Serialized 14 byte log entry
0808 01:11:11.204714 (+ 192us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:11.204788 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:12.023163 (+818375us) raft_consensus.cc:1671] finished
0808 01:11:12.023190 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:12.023353 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45062,"prepare.queue_time_us":38314,"prepare.run_cpu_time_us":125,"prepare.run_wall_time_us":7980,"raft.queue_time_us":20874,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":10507,"thread_start_us":244882,"threads_started":1,"wal-append.queue_time_us":360873}
I20260808 01:11:11.956879 4319 consensus_queue.cc:260] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:12.087494 4483 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:12.138201 1604 raft_consensus.cc:2396] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:11:12.034147 4273 consensus_queue.cc:260] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:12.149749 1596 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.744s user 0.032s sys 0.103s
W20260808 01:11:12.358259 1596 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.953s user 0.034s sys 0.117s
W20260808 01:11:12.354727 1566 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.009s sys 0.021s
W20260808 01:11:12.359158 1566 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.516s user 0.009s sys 0.021s
I20260808 01:11:12.124655 4550 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:12.281118 1594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2581) took 1136 ms. Trace:
I20260808 01:11:12.371755 1594 rpcz_store.cc:276] 0808 01:11:11.145060 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:11.145222 (+ 162us) service_pool.cc:227] Handling call
0808 01:11:12.280884 (+1135662us) spinlock_profiling.cc:243] Waited 626 ms on lock 0x7b540024f9d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:12.281041 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1439944064}
I20260808 01:11:12.302103 1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2579) took 1158 ms. Trace:
I20260808 01:11:12.123782 4534 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:12.369794 1568 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.021s sys 0.041s
I20260808 01:11:12.302870 1438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55040 (request call id 2128) took 1008 ms. Trace:
I20260808 01:11:12.390815 1438 rpcz_store.cc:276] 0808 01:11:11.294434 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:11.294666 (+ 232us) service_pool.cc:227] Handling call
0808 01:11:12.302716 (+1008050us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b540002e4d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:12.302816 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":488424192}
I20260808 01:11:12.374039 1599 rpcz_store.cc:276] 0808 01:11:11.143773 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:11.144298 (+ 525us) service_pool.cc:227] Handling call
0808 01:11:11.144647 (+ 349us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:11.931815 (+787168us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:12.301814 (+369999us) spinlock_profiling.cc:243] Waited 768 ms on lock 0x7b540018d8d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:12.302030 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1767107456}
W20260808 01:11:12.265304 4477 consensus_meta.cc:287] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 1.198s user 0.009s sys 0.006s
W20260808 01:11:12.390247 1568 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.549s user 0.022s sys 0.042s
W20260808 01:11:12.395471 1442 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.661s user 0.008s sys 0.020s
I20260808 01:11:12.143743 4433 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:12.299340 4378 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:11:12.141593 4453 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.023s sys 0.037s
I20260808 01:11:12.294654 4521 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:12.256171 4445 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:12.189419 4421 raft_consensus.cc:3037] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:12.293329 4544 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:12.283423 4537 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:12.227946 4491 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:12.409893 2205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 865) took 1515 ms. Trace:
I20260808 01:11:12.444806 2205 rpcz_store.cc:276] 0808 01:11:10.894429 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:11.110652 (+216223us) service_pool.cc:227] Handling call
0808 01:11:11.111035 (+ 383us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:11.249754 (+138719us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:11.249788 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:11.250270 (+ 482us) log.cc:842] Serialized 26 byte log entry
0808 01:11:11.336371 (+ 86101us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:11.336431 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:12.039080 (+702649us) spinlock_profiling.cc:243] Waited 4.12 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:12.409434 (+370354us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54007d05d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:12.409526 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:12.409595 (+ 69us) raft_consensus.cc:1671] finished
0808 01:11:12.409616 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:12.409813 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5283,"raft.queue_time_us":49397,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":26704,"spinlock_wait_cycles":282472064,"thread_start_us":99838,"threads_started":2,"wal-append.queue_time_us":223845}
I20260808 01:11:12.410260 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:12.448537 1447 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:12.374290 4483 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.454269 4483 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:12.448094 4540 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:12.439256 4099 raft_consensus.cc:2726] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7.
I20260808 01:11:12.457278 4540 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:12.392560 4547 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:12.401146 4566 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:12.401480 4477 raft_consensus.cc:2726] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:11:12.404330 4453 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.906s user 0.024s sys 0.038s
I20260808 01:11:12.423131 4525 raft_consensus.cc:3032] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:12.427246 2209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 869) took 1339 ms. Trace:
I20260808 01:11:12.489634 4525 raft_consensus.cc:729] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:12.490526 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:12.489727 2209 rpcz_store.cc:276] 0808 01:11:11.088306 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:11.179190 (+ 90884us) service_pool.cc:227] Handling call
0808 01:11:11.208848 (+ 29658us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:11.316292 (+107444us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:11.316339 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 01:11:11.353153 (+ 36814us) log.cc:842] Serialized 52 byte log entry
0808 01:11:11.960428 (+607275us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:11.960470 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:11.960542 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:12.426903 (+466361us) raft_consensus.cc:1671] finished
0808 01:11:12.426924 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:12.427158 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11001,"raft.queue_time_us":37131,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":9644,"thread_start_us":653237,"threads_started":2,"wal-append.queue_time_us":937726}
I20260808 01:11:12.491525 1599 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:12.492467 1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 579) took 1118 ms. Trace:
I20260808 01:11:12.492844 1604 rpcz_store.cc:276] 0808 01:11:11.374206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:11.375542 (+ 1336us) service_pool.cc:227] Handling call
0808 01:11:12.492283 (+1116741us) spinlock_profiling.cc:243] Waited 761 ms on lock 0x7b54003908d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:12.492413 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1750706432}
I20260808 01:11:12.493852 1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2582) took 1335 ms. Trace:
I20260808 01:11:12.494450 1596 rpcz_store.cc:276] 0808 01:11:11.158744 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:11.159451 (+ 707us) service_pool.cc:227] Handling call
0808 01:11:11.171772 (+ 12321us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:11.328437 (+156665us) raft_consensus.cc:1520] Early marking committed up to index 7
0808 01:11:11.381234 (+ 52797us) log.cc:842] Serialized 14 byte log entry
0808 01:11:12.304434 (+923200us) spinlock_profiling.cc:243] Waited 6.2 ms on lock 0x7b18003125e0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349b17764 00007fe349b1ab11 00007fe349b1aac9 00007fe349b1aa59 00007fe349b1aa21 00007fe349b19d1d 00007fe352d7b204 00007fe352d7b039 00007fe349af9dc6
0808 01:11:12.359365 (+ 54931us) log.cc:842] Serialized 14 byte log entry
0808 01:11:12.359730 (+ 365us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:12.359788 (+ 58us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:12.359873 (+ 85us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:12.493622 (+133749us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b54001e9e54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:12.493675 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:12.493799 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6073,"prepare.queue_time_us":162388,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":3712,"raft.queue_time_us":107966,"raft.run_cpu_time_us":374,"raft.run_wall_time_us":24700,"spinlock_wait_cycles":373967488,"thread_start_us":969748,"threads_started":3,"wal-append.queue_time_us":343265}
I20260808 01:11:12.432751 1438 raft_consensus.cc:1226] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:12.434055 4273 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:11:12.491346 4525 consensus_queue.cc:260] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:12.501533 4525 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:11:12.436343 4387 consensus_meta.cc:287] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 1.167s user 0.006s sys 0.010s
I20260808 01:11:12.437176 4352 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:12.413269 4488 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:12.479462 4401 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:12.509269 4387 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.509976 4401 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.396539 4534 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:12.441128 4289 consensus_meta.cc:287] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.667s user 0.013s sys 0.000s
I20260808 01:11:12.552927 4289 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:12.395924 1442 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.661s user 0.009s sys 0.020s
I20260808 01:11:12.126971 4415 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.487003 1517 raft_consensus.cc:1226] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:12.480976 4566 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.441087 4521 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:12.577698 4521 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:12.579207 4566 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:12.584514 4289 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:12.585999 4481 consensus_meta.cc:287] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.646s user 0.005s sys 0.009s
I20260808 01:11:12.228768 3748 consensus_queue.cc:260] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:12.507206 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:12.472923 4496 consensus_meta.cc:287] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.642s user 0.004s sys 0.009s
W20260808 01:11:12.398335 4416 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.009s sys 0.022s
I20260808 01:11:12.507772 1568 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:12.534924 4401 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:12.591475 4416 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.753s user 0.009s sys 0.022s
I20260808 01:11:12.536932 4319 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:12.537947 4387 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:12.540913 4540 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:12.545805 4413 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:12.595686 4533 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:12.480659 4547 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.610368 4547 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:12.560159 2203 raft_consensus.cc:1226] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:12.444113 4480 consensus_queue.cc:260] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:12.480232 4445 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.560304 4565 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.620723 4445 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:12.621713 4565 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:12.622725 2203 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 875) took 1202 ms. Trace:
I20260808 01:11:12.551349 4567 consensus_queue.cc:237] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.569566 1517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2172) took 1607 ms. Trace:
I20260808 01:11:12.623986 1517 rpcz_store.cc:276] 0808 01:11:10.962548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:10.962739 (+ 191us) service_pool.cc:227] Handling call
0808 01:11:10.963085 (+ 346us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:12.568693 (+1605608us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:12.569290 (+ 597us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b540012d754. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:12.569499 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3503932288}
I20260808 01:11:12.488262 4491 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.586619 4481 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.588655 2206 raft_consensus.cc:1226] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:12.590487 4415 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:12.591264 1604 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:12.570173 4534 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:12.596801 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 89105.5us
I20260808 01:11:12.610999 4401 raft_consensus.cc:2781] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:12.596833 4544 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:12.550042 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:12.597707 4533 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.623211 2203 rpcz_store.cc:276] 0808 01:11:11.420479 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:12.017717 (+597238us) service_pool.cc:227] Handling call
0808 01:11:12.018140 (+ 423us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:12.621690 (+603550us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:12.622416 (+ 726us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b540059fbd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:12.622654 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1235689216}
I20260808 01:11:12.655766 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 872) took 1422 ms. Trace:
I20260808 01:11:12.657269 2206 rpcz_store.cc:276] 0808 01:11:11.233661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:11.848352 (+614691us) service_pool.cc:227] Handling call
0808 01:11:11.893058 (+ 44706us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:12.639604 (+746546us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:12.655466 (+ 15862us) spinlock_profiling.cc:243] Waited 695 ms on lock 0x7b54004c54d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:12.655691 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1599167360}
W20260808 01:11:12.671255 1520 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.827s user 0.003s sys 0.030s
W20260808 01:11:12.671864 1520 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s user 0.003s sys 0.031s
I20260808 01:11:12.656708 4401 raft_consensus.cc:686] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
W20260808 01:11:12.677452 4555 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.759s user 0.021s sys 0.031s
W20260808 01:11:12.682324 4528 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.289s user 0.009s sys 0.026s
I20260808 01:11:12.673808 4461 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:12.656508 4544 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.682021 4543 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:12.699584 3748 raft_consensus.cc:3037] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:11:12.697261 2202 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s user 0.006s sys 0.011s
I20260808 01:11:12.686852 4480 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:11:12.701546 4555 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.788s user 0.021s sys 0.032s
W20260808 01:11:12.681661 4189 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.771s user 0.004s sys 0.014s
W20260808 01:11:12.735307 2202 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s user 0.006s sys 0.011s
W20260808 01:11:12.728556 4291 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.768s user 0.039s sys 0.107s
W20260808 01:11:12.702622 4528 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.314s user 0.009s sys 0.027s
W20260808 01:11:12.743764 4291 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.790s user 0.039s sys 0.107s
I20260808 01:11:12.733098 4413 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:12.734241 4568 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:12.760980 4433 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:12.743276 4189 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.843s user 0.005s sys 0.014s
I20260808 01:11:12.763589 4481 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:12.736970 4534 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:12.728621 4461 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:12.758216 4539 consensus_meta.cc:287] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.628s user 0.009s sys 0.012s
W20260808 01:11:12.767082 4479 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.848s user 0.015s sys 0.074s
I20260808 01:11:12.792397 4574 raft_consensus.cc:3032] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:11:12.755631 4269 consensus_queue.cc:1046] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260808 01:11:12.733803 4543 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:12.821071 4491 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:12.789891 4568 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.802191 4534 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.819342 4481 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:11:12.829144 4479 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.910s user 0.015s sys 0.074s
I20260808 01:11:12.773729 4413 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.789306 4401 consensus_queue.cc:237] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:12.827847 4433 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:12.830885 4574 raft_consensus.cc:729] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:12.878202 4533 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:12.834093 3772 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.712s user 0.004s sys 0.016s
W20260808 01:11:12.885891 4295 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.003s sys 0.011s
W20260808 01:11:12.937407 1408 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.047s user 0.009s sys 0.024s
W20260808 01:11:12.973950 1408 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.116s user 0.009s sys 0.025s
W20260808 01:11:12.974036 4295 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s user 0.004s sys 0.011s
W20260808 01:11:12.974347 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 1.1253e+06us
I20260808 01:11:12.938798 1440 raft_consensus.cc:1226] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260808 01:11:12.947798 1409 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.017s user 0.007s sys 0.009s
W20260808 01:11:12.978168 1409 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.055s user 0.007s sys 0.010s
W20260808 01:11:12.939142 1445 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.039s user 0.014s sys 0.016s
W20260808 01:11:12.978767 1445 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.079s user 0.014s sys 0.017s
I20260808 01:11:12.882017 4481 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:12.948799 1490 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.075s user 0.016s sys 0.015s
W20260808 01:11:12.980464 1490 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.107s user 0.016s sys 0.015s
W20260808 01:11:12.980899 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 1.10767e+06us
I20260808 01:11:12.982014 1490 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
W20260808 01:11:12.948113 1410 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.090s user 0.017s sys 0.057s
W20260808 01:11:12.983969 1410 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.126s user 0.018s sys 0.057s
W20260808 01:11:12.984336 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 1.12798e+06us
I20260808 01:11:12.984161 4433 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:12.985613 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:12.985697 4433 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:12.987433 1446 raft_consensus.cc:2454] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:12.990036 2203 raft_consensus.cc:1226] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 8. Current term is 9. Ops: []
I20260808 01:11:12.990168 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:12.991279 1523 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:12.992494 4367 raft_consensus.cc:2781] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:11:12.992872 1446 raft_consensus.cc:1226] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 7. Current term is 8. Ops: []
I20260808 01:11:12.993582 4367 raft_consensus.cc:686] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:12.994947 1448 raft_consensus.cc:1226] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 6. Current term is 7. Ops: []
W20260808 01:11:12.959323 2171 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.017s user 0.013s sys 0.010s
W20260808 01:11:12.996564 2171 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.054s user 0.013s sys 0.010s
I20260808 01:11:12.999238 2206 raft_consensus.cc:1261] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 3 index: 6. (term mismatch)
I20260808 01:11:12.970445 4179 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:12.977046 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:12.968571 4057 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.006s sys 0.026s
I20260808 01:11:13.001032 4179 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:13.001967 4057 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s user 0.006s sys 0.027s
I20260808 01:11:12.936796 1442 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 691) took 1202 ms. Trace:
I20260808 01:11:12.966161 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:13.005949 2205 raft_consensus.cc:1261] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEARNER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260808 01:11:13.004073 1442 rpcz_store.cc:276] 0808 01:11:11.733027 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:11.733194 (+ 167us) service_pool.cc:227] Handling call
0808 01:11:11.733570 (+ 376us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:11.734013 (+ 443us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:11.734050 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:11.734632 (+ 582us) log.cc:842] Serialized 26 byte log entry
0808 01:11:12.561841 (+827209us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:12.561894 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:12.561990 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:12.934707 (+372717us) raft_consensus.cc:1671] finished
0808 01:11:12.934857 (+ 150us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:12.935633 (+ 776us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11328,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":8215,"thread_start_us":826777,"threads_started":1,"wal-append.queue_time_us":1009311}
I20260808 01:11:12.995505 4367 consensus_queue.cc:237] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.011333 1440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 682) took 2528 ms. Trace:
I20260808 01:11:13.020840 1440 rpcz_store.cc:276] 0808 01:11:10.482527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:10.499428 (+ 16901us) service_pool.cc:227] Handling call
0808 01:11:10.515373 (+ 15945us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:12.977990 (+2462617us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:13.010821 (+ 32831us) spinlock_profiling.cc:243] Waited 2.42 s on lock 0x7b54001beed4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:13.011216 (+ 395us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5570191616}
I20260808 01:11:13.011548 4293 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:13.023125 4293 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.015493 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:13.024893 2205 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
W20260808 01:11:12.969817 3772 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.848s user 0.004s sys 0.016s
I20260808 01:11:13.026513 4293 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:13.028640 1439 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55040 (request call id 2137) took 1445 ms (client timeout 1527 ms). Trace:
I20260808 01:11:13.029178 4179 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:13.029078 1439 rpcz_store.cc:269] 0808 01:11:11.583438 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:11.586670 (+ 3232us) service_pool.cc:227] Handling call
0808 01:11:13.028228 (+1441558us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b540009cc54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:13.028459 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2952141184}
W20260808 01:11:12.952095 1489 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.090s user 0.009s sys 0.009s
W20260808 01:11:12.947306 1568 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.527s (SENT) [suppressed 17 similar messages]
I20260808 01:11:13.017855 4413 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:13.018177 4433 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:13.017928 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:12.999706 2206 pending_rounds.cc:76] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd: Aborting all ops after (but not including) 5
I20260808 01:11:13.018810 4291 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:12.958894 1520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2181) took 1126 ms. Trace:
W20260808 01:11:12.959116 1567 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.099s user 0.005s sys 0.011s
I20260808 01:11:13.019667 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:11:13.051407 1489 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.190s user 0.009s sys 0.009s
I20260808 01:11:13.027544 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:12.982640 4481 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:13.014083 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:13.019938 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:13.040828 2206 pending_rounds.cc:97] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd: Aborting uncommitted NO_OP operation due to leader change: 4.6
W20260808 01:11:13.068825 1567 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.208s user 0.006s sys 0.011s
I20260808 01:11:13.064852 2206 raft_consensus.cc:2866] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:13.070806 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:13.082932 4491 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:13.084159 4598 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:13.084093 4491 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:13.078465 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 1.20741e+06us
W20260808 01:11:13.091152 4378 consensus_meta.cc:287] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.612s user 0.010s sys 0.010s
W20260808 01:11:13.094151 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 1.22955e+06us
I20260808 01:11:13.098589 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:13.104749 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:13.051301 1520 rpcz_store.cc:276] 0808 01:11:11.831942 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:11.832123 (+ 181us) service_pool.cc:227] Handling call
0808 01:11:11.832575 (+ 452us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:11.837374 (+ 4799us) raft_consensus.cc:1520] Early marking committed up to index 5
0808 01:11:11.837403 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:11.844437 (+ 7034us) log.cc:842] Serialized 26 byte log entry
0808 01:11:12.672908 (+828471us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:12.672965 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:12.673063 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:12.958339 (+285276us) raft_consensus.cc:1671] finished
0808 01:11:12.958379 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:12.958661 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4370,"raft.queue_time_us":24219,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":43165,"thread_start_us":827927,"threads_started":1,"wal-append.queue_time_us":1031347}
I20260808 01:11:13.110551 2200 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
W20260808 01:11:13.122417 2204 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.816s user 0.022s sys 0.039s
W20260808 01:11:13.123294 2204 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.818s user 0.023s sys 0.039s
I20260808 01:11:13.085387 4598 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:13.124315 4558 raft_consensus.cc:3032] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:11:12.902789 4534 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:12.899338 4568 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:13.138142 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:13.140381 4406 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:13.141116 4406 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.142014 4598 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:12.895679 4544 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:12.902148 4543 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:13.117704 3772 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:13.107014 2203 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:13.156260 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:13.147856 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:13.129797 4558 raft_consensus.cc:729] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
W20260808 01:11:13.178959 4435 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.042s user 0.010s sys 0.036s
I20260808 01:11:13.184381 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:13.129635 4545 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:11:13.180016 4417 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.035s user 0.087s sys 0.102s
W20260808 01:11:13.197678 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 49569.9us
W20260808 01:11:13.179790 4435 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.088s user 0.010s sys 0.036s
I20260808 01:11:12.897053 4574 consensus_queue.cc:260] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.116946 4291 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:13.150583 4597 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:13.206209 2205 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:13.204442 1438 raft_consensus.cc:2454] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:11:13.204275 3772 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:13.204330 4417 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.110s user 0.087s sys 0.102s
I20260808 01:11:13.180018 4477 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:13.217974 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:13.222362 1442 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:13.226469 4586 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:13.227002 4461 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:11:13.227376 4572 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.002s sys 0.049s
W20260808 01:11:13.236581 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 289439us
I20260808 01:11:13.254556 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:12.902994 4550 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:13.244441 4572 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.690s user 0.002s sys 0.050s
I20260808 01:11:13.292440 4295 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:13.299429 1489 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:13.291923 4491 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:13.328548 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:13.336905 4412 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:13.337256 4595 log.cc:924] Time spent T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20260808 01:11:13.338191 4412 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:13.336995 4416 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:13.340240 4295 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.341396 4416 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.347759 2202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 663) took 1999 ms. Trace:
I20260808 01:11:13.348582 4586 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:13.351073 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 880) took 1553 ms. Trace:
I20260808 01:11:13.351173 2209 raft_consensus.cc:1226] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:13.355758 4598 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:13.356822 4598 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.358960 4293 consensus_queue.cc:1035] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:11:13.362592 4534 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:13.363472 4534 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.365123 1568 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:13.366297 4534 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:13.368974 3772 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:13.371474 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:13.373703 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:13.375468 1442 raft_consensus.cc:2379] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 4: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:13.351588 2204 rpcz_store.cc:276] 0808 01:11:11.798182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:12.149617 (+351435us) service_pool.cc:227] Handling call
0808 01:11:12.266549 (+116932us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:12.266779 (+ 230us) raft_consensus.cc:1520] Early marking committed up to index 7
0808 01:11:13.124087 (+857308us) log.cc:842] Serialized 14 byte log entry
0808 01:11:13.350368 (+226281us) log.cc:842] Serialized 14 byte log entry
0808 01:11:13.350554 (+ 186us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:13.350617 (+ 63us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:13.350726 (+ 109us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:13.350857 (+ 131us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:13.351011 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":97901,"prepare.queue_time_us":952916,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":31820,"thread_start_us":894794,"threads_started":2}
I20260808 01:11:13.376391 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:13.377506 4550 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:13.377985 4412 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:13.384615 2205 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
W20260808 01:11:13.391772 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 62549.3us
I20260808 01:11:13.343590 4295 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:13.339004 4477 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.348330 2202 rpcz_store.cc:276] 0808 01:11:11.348187 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:11.917463 (+569276us) service_pool.cc:227] Handling call
0808 01:11:11.933670 (+ 16207us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:11.939760 (+ 6090us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:11.939798 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 01:11:11.956961 (+ 17163us) log.cc:842] Serialized 52 byte log entry
0808 01:11:12.742948 (+785987us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:13.124551 (+381603us) log.cc:842] Serialized 14 byte log entry
0808 01:11:13.346452 (+221901us) log.cc:842] Serialized 14 byte log entry
0808 01:11:13.346872 (+ 420us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:13.347085 (+ 213us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:13.347202 (+ 117us) raft_consensus.cc:1671] finished
0808 01:11:13.347234 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:13.347474 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":221788,"prepare.queue_time_us":583322,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":85,"raft.queue_time_us":326417,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":373,"thread_start_us":1077453,"threads_started":2,"wal-append.queue_time_us":920476}
I20260808 01:11:13.256572 4597 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.377401 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:13.380743 4416 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:13.303380 4545 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.442765 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:13.444415 4545 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:13.443564 2204 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:13.445084 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:13.434077 4596 raft_consensus.cc:2781] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:13.472220 4596 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:13.472934 4596 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:13.434790 4558 consensus_queue.cc:260] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.476687 4558 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:11:13.480087 4412 consensus_queue.cc:1159] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" responder_term: 7 status { last_received { term: 6 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260808 01:11:13.435918 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 173017us
I20260808 01:11:13.436834 2203 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:13.439973 1443 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 8.
I20260808 01:11:13.441334 4550 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:13.441821 4598 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:13.445401 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:13.449240 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:13.490139 1568 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:11:13.433799 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 211029us
I20260808 01:11:13.451318 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:13.455632 4406 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:13.451349 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:13.457473 4591 raft_consensus.cc:2726] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:13.459148 4477 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:13.461519 4597 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:13.461726 4553 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:13.465533 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:13.467772 4586 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:13.491340 2202 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:13.527627 2206 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:11:13.556802 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 63698.7us
I20260808 01:11:13.556432 1442 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:13.559508 2205 raft_consensus.cc:2396] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 01:11:13.568907 2200 raft_consensus.cc:2354] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3.
I20260808 01:11:13.576577 1568 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:13.584342 4291 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:13.588308 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 351372us
I20260808 01:11:13.573959 4401 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:13.596072 4606 log.cc:924] Time spent T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23: Append to log took a long time: real 0.167s user 0.004s sys 0.000s
I20260808 01:11:13.574447 4550 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:13.598292 1410 connection.cc:232] RPC call timeout handler was delayed by -1.0001084220129997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17420000000000002s for a response
I20260808 01:11:13.580027 4295 consensus_queue.cc:1046] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260808 01:11:13.614660 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:13.615703 2205 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:13.631754 4597 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:13.632114 4291 consensus_queue.cc:1046] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260808 01:11:13.633092 1447 consensus_meta.cc:287] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 1.129s user 0.019s sys 0.010s
I20260808 01:11:13.633666 1447 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:13.633006 4597 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:13.586202 4553 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.607434 4477 consensus_queue.cc:1046] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260808 01:11:13.640480 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:13.641660 4480 consensus_meta.cc:287] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.883s user 0.015s sys 0.007s
W20260808 01:11:13.642647 4610 log.cc:924] Time spent T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.208s user 0.003s sys 0.005s
I20260808 01:11:13.644239 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:13.645881 1447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55040 (request call id 2142) took 1236 ms. Trace:
I20260808 01:11:13.647145 2205 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:13.628762 4406 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:13.646260 1447 rpcz_store.cc:276] 0808 01:11:12.409777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:12.409946 (+ 169us) service_pool.cc:227] Handling call
0808 01:11:13.638274 (+1228328us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:13.651084 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:13.654297 2209 raft_consensus.cc:2396] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 9 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260808 01:11:13.638571 4269 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:11:13.658489 4571 raft_consensus.cc:2781] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:13.659016 4571 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:13.658612 4406 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.659492 4571 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:13.660004 4553 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:13.657537 2202 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:13.646663 4598 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:13.667047 4598 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:13.677493 4597 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:13.677976 4517 raft_consensus.cc:3032] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:13.678462 4517 raft_consensus.cc:729] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:13.678805 4550 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:13.679967 4589 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:13.680586 4589 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:13.683079 4589 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:13.679500 4517 consensus_queue.cc:260] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.685752 1410 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:13.664074 4269 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:13.689716 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:13.644291 4401 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:13.694185 4269 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:13.713182 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:13.713968 2207 raft_consensus.cc:2362] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7. Re-sending same reply.
I20260808 01:11:13.720392 1487 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:13.710671 2209 raft_consensus.cc:2354] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 2. Current term is 3.
I20260808 01:11:13.720575 2201 raft_consensus.cc:3037] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEARNER]: Advancing to term 5
I20260808 01:11:13.725381 4598 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:13.706568 4295 raft_consensus.cc:2781] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:13.703992 4550 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:13.733009 2206 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:13.735410 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:13.746881 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:13.732545 4295 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:13.756114 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:13.745107 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 145732us
I20260808 01:11:13.757783 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:13.758414 1443 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 8.
I20260808 01:11:13.761847 1604 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:13.762169 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:13.735628 4534 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:13.712704 4586 consensus_queue.cc:1046] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.035s
I20260808 01:11:13.743284 4421 consensus_queue.cc:1046] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.082s
I20260808 01:11:13.754647 4517 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:13.776615 1442 raft_consensus.cc:1261] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:11:13.770807 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:13.763370 4295 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:13.783234 4568 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:13.787529 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:13.747730 4544 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:13.767006 4598 raft_consensus.cc:2726] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:13.783910 4568 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:13.800216 1489 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:13.799899 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:13.764700 4550 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.802201 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:13.804177 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:13.804625 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:13.804920 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:13.754984 4588 consensus_queue.cc:1035] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.127s
I20260808 01:11:13.793810 4378 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:13.819205 1445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2136) took 2236 ms. Trace:
I20260808 01:11:13.788738 4534 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.823643 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:13.826215 2204 raft_consensus.cc:2396] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260808 01:11:13.829030 2207 raft_consensus.cc:1226] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 7. Current term is 8. Ops: []
I20260808 01:11:13.829509 1520 raft_consensus.cc:2379] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 4: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:13.825214 2209 raft_consensus.cc:2454] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:11:13.831691 1408 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:13.814064 4480 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:13.820212 1445 rpcz_store.cc:276] 0808 01:11:11.582866 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:11.680803 (+ 97937us) service_pool.cc:227] Handling call
0808 01:11:11.719771 (+ 38968us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:11.844997 (+125226us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 01:11:12.979447 (+1134450us) log.cc:842] Serialized 14 byte log entry
0808 01:11:13.242675 (+263228us) log.cc:842] Serialized 14 byte log entry
0808 01:11:13.292965 (+ 50290us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:13.303878 (+ 10913us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:13.303960 (+ 82us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:13.478960 (+175000us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b4800251958. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3518e67f1 00007fe3518dd8e7 00007fe3518dd77a 00007fe3519d195f 00007fe3519d1627 00007fe3519e5b2c 00007fe3519e5ad8 00007fe3519e5a58 00007fe3519e5a13 00007fe3519e4cbc 00007fe350966be6
0808 01:11:13.605692 (+126732us) spinlock_profiling.cc:243] Waited 5.95 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:13.812113 (+206421us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54001bf8d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:13.812148 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:13.819146 (+ 6998us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21423,"prepare.queue_time_us":887316,"prepare.run_cpu_time_us":121,"prepare.run_wall_time_us":5305,"raft.queue_time_us":88566,"raft.run_cpu_time_us":1946,"raft.run_wall_time_us":180731,"spinlock_wait_cycles":326476544,"thread_start_us":1244186,"threads_started":2,"wal-append.queue_time_us":453745}
I20260808 01:11:13.825103 4057 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:13.830535 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:13.841845 2172 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:13.843091 2172 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:13.844097 4057 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:13.846312 4057 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:13.840299 1521 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:13.808137 4544 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.827009 4598 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:13.829325 4401 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:13.835420 4586 raft_consensus.cc:2781] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:13.838145 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:13.857386 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:11:13.859068 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 81372.2us
I20260808 01:11:13.860240 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:13.858143 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:13.861692 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:13.862812 1517 raft_consensus.cc:1226] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 8. Current term is 9. Ops: []
I20260808 01:11:13.863133 4596 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:13.864603 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:11:13.867080 4612 log.cc:924] Time spent T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.432s user 0.000s sys 0.010s
I20260808 01:11:13.867784 4596 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:13.867857 1603 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260808 01:11:13.852596 4481 raft_consensus.cc:2781] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:13.858379 4586 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:13.838485 4378 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:13.863718 4571 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:13.867942 4461 raft_consensus.cc:2781] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:13.885308 4481 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:13.899011 1596 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:13.900543 1604 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:13.890573 4586 raft_consensus.cc:3037] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:13.909644 1445 raft_consensus.cc:1261] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260808 01:11:13.914537 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:13.903152 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:13.896129 4461 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:13.922161 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:13.902192 4481 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:13.933738 1594 raft_consensus.cc:2354] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 6. Current term is 7.
I20260808 01:11:13.932046 1489 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:13.883886 4480 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:13.931905 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:13.941584 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:13.942828 1603 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:13.940371 4589 raft_consensus.cc:3032] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:13.945354 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:13.954741 1447 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 11.
I20260808 01:11:13.952571 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:13.924676 4461 raft_consensus.cc:3037] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:13.953924 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:13.954767 1443 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:13.953106 4589 raft_consensus.cc:729] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:13.957036 1440 raft_consensus.cc:2454] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:11:13.936640 4550 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:13.938256 4568 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:13.970026 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:13.970608 1525 raft_consensus.cc:1226] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:13.970840 4480 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:13.976389 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:13.983038 1603 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 9.
I20260808 01:11:13.984783 4534 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:13.995425 1594 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:11:13.997442 1487 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:13.981362 4591 raft_consensus.cc:2781] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:13.998332 4477 consensus_queue.cc:1035] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:11:13.999183 1487 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:13.999976 1438 raft_consensus.cc:2454] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:11:14.000078 4625 raft_consensus.cc:2781] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:14.000425 4477 raft_consensus.cc:2781] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:14.000769 4625 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:14.000905 4477 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:14.001233 4625 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:13.998684 4591 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:14.001824 4591 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:13.970089 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.004298 1487 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
W20260808 01:11:14.005095 4626 log.cc:924] Time spent T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260808 01:11:14.005414 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:14.005671 4544 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:14.006083 1602 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:11:14.006534 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 106210us
I20260808 01:11:14.005463 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:14.008666 1603 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:14.001370 4477 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:14.012122 4597 consensus_queue.cc:1046] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:14.013980 4597 raft_consensus.cc:3032] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:14.014413 4597 raft_consensus.cc:729] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:14.014002 4574 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:14.016571 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.017553 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:14.021253 4568 consensus_queue.cc:1035] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:11:14.022022 1594 raft_consensus.cc:1261] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:11:14.023098 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.038359 4099 raft_consensus.cc:3032] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:14.038888 4099 raft_consensus.cc:729] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:14.039808 4099 consensus_queue.cc:260] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.041323 4099 raft_consensus.cc:3037] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:14.041921 4421 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 01:11:14.023452 1604 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:14.024421 3772 raft_consensus.cc:3032] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:11:14.024977 1600 raft_consensus.cc:1261] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:11:14.025317 4293 raft_consensus.cc:3032] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:13.963083 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.014287 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.034282 4416 raft_consensus.cc:3032] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:14.045415 1519 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:14.081775 1599 raft_consensus.cc:1226] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:14.085345 2201 raft_consensus.cc:1261] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEARNER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260808 01:11:14.082468 4293 raft_consensus.cc:729] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:13.991957 1568 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:14.088900 4293 consensus_queue.cc:260] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.096130 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.094769 4417 consensus_queue.cc:1046] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.016s
I20260808 01:11:14.082472 3772 raft_consensus.cc:729] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:14.110355 4589 consensus_queue.cc:260] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.122047 1438 raft_consensus.cc:2379] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:14.123521 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:14.126767 1438 raft_consensus.cc:1226] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:14.128302 4571 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:14.129732 1438 raft_consensus.cc:1226] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
W20260808 01:11:14.130100 4634 log.cc:924] Time spent T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.153s user 0.000s sys 0.000s
I20260808 01:11:14.095907 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:11:14.132680 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 140821us
I20260808 01:11:14.054008 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:14.134841 1568 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:14.140514 1568 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:11:14.142675 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 272565us
W20260808 01:11:14.147192 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 143014us
I20260808 01:11:14.082377 4416 raft_consensus.cc:729] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:14.150753 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.150211 2172 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:14.150315 1602 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:11:14.149343 1518 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:14.156775 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:14.160413 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.161605 1445 raft_consensus.cc:2379] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 11: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:14.163631 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.145123 4571 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:11:14.182448 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.184432 2201 raft_consensus.cc:1261] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:11:14.159909 2208 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:48980
I20260808 01:11:14.188467 2208 raft_consensus.cc:595] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:14.188849 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.189680 1602 raft_consensus.cc:2379] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 6: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:14.190554 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.015571 4597 consensus_queue.cc:260] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.196915 4597 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:14.204387 1565 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:14.146992 4629 raft_consensus.cc:2781] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:14.191468 1602 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
W20260808 01:11:14.218477 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 70409.8us
W20260808 01:11:14.221038 4620 log.cc:924] Time spent T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.558s user 0.003s sys 0.000s
I20260808 01:11:14.224782 1565 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:14.217067 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.226193 1438 raft_consensus.cc:2379] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 3: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:14.229543 4295 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:14.205919 4293 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:14.232663 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.235865 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:14.220286 4269 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.237247 4412 raft_consensus.cc:2726] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:14.147287 3772 consensus_queue.cc:260] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.238816 4269 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:14.239167 1565 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:14.239917 4589 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:14.240267 1565 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:14.241863 4098 raft_consensus.cc:2781] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:14.242277 4098 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:14.242640 4098 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:14.162815 4491 raft_consensus.cc:2781] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:14.241876 1603 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:14.248086 4416 consensus_queue.cc:260] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.246634 4491 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:14.252556 4491 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:14.216532 4629 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:14.253958 4629 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:14.249001 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.246817 4568 raft_consensus.cc:2781] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:14.258138 1600 raft_consensus.cc:2454] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:11:14.258489 4568 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:14.251431 1487 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:14.258767 1603 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:14.259224 1440 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:11:14.273094 1440 raft_consensus.cc:595] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:14.273569 1440 raft_consensus.cc:603] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:11:14.257696 4416 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:14.270669 1594 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:14.263446 1604 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 8.
I20260808 01:11:14.276602 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.176797 4571 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:14.279522 1443 raft_consensus.cc:2454] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
W20260808 01:11:14.282668 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 132541us
I20260808 01:11:14.285097 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.285934 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.287791 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:11:14.289122 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 48938.1us
I20260808 01:11:14.291635 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:14.292651 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 73447.2us
I20260808 01:11:14.292927 4593 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:14.294478 4295 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:14.295375 1600 raft_consensus.cc:2379] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 8: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:14.292944 1565 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:14.298259 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:14.300971 1525 raft_consensus.cc:1261] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:11:14.301421 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.301640 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.303227 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.287418 4571 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:11:14.304054 4545 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:11:14.304746 4635 consensus_queue.cc:1035] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:14.308944 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:14.312517 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.313246 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.313915 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.315022 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:14.337659 1603 raft_consensus.cc:2354] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 8. Current term is 9.
I20260808 01:11:14.251560 4480 raft_consensus.cc:2781] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:14.275365 4596 raft_consensus.cc:2781] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:14.275729 4568 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:14.277374 4378 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:14.320762 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.277539 4406 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:14.325263 1604 raft_consensus.cc:2379] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 7: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:14.328073 1594 raft_consensus.cc:2396] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260808 01:11:14.372236 1596 raft_consensus.cc:2454] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:14.332283 1487 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:14.333830 2171 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:14.317651 4544 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:14.339371 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.340123 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:14.379222 1446 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:14.340366 2208 raft_consensus.cc:1226] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:14.379077 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.341382 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.342386 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.349268 4591 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:14.330977 4625 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.352315 4461 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.335873 4545 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:14.358615 4593 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.359978 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.317981 3772 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:14.361866 4480 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:14.419544 4480 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:14.434486 4099 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:14.251173 4098 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.373066 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.437466 1600 raft_consensus.cc:2454] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 2.
I20260808 01:11:14.329804 2205 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:48980
I20260808 01:11:14.340545 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.438793 2205 raft_consensus.cc:595] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:14.439383 2205 raft_consensus.cc:603] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:11:14.380476 1603 raft_consensus.cc:2379] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 9: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:14.394750 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.397393 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.357339 4477 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.399294 1443 raft_consensus.cc:2379] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:14.448308 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.362129 4269 raft_consensus.cc:2781] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:14.361603 4553 raft_consensus.cc:2781] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:14.455947 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:14.456929 1442 raft_consensus.cc:3037] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:14.400830 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.402433 4544 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.402541 4293 consensus_queue.cc:1035] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260808 01:11:14.414016 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.417444 4593 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:14.418699 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:14.466745 1517 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:14.466814 1448 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:14.420953 4481 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:14.422055 4558 consensus_meta.cc:287] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.945s user 0.014s sys 0.006s
I20260808 01:11:14.402727 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.403062 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.425191 1568 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:14.331736 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.408046 1516 raft_consensus.cc:2454] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
W20260808 01:11:14.408676 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 118162us
I20260808 01:11:14.408741 4378 raft_consensus.cc:2781] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:14.428895 4386 raft_consensus.cc:3032] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:11:14.412524 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:14.412830 4517 consensus_meta.cc:287] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.622s user 0.011s sys 0.011s
I20260808 01:11:14.426957 1519 raft_consensus.cc:1226] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:14.433425 4596 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:14.435146 4099 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:14.438740 1599 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:14.404997 4545 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:14.408154 4461 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:14.457847 1518 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:14.464999 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.465873 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.433463 4591 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:14.470842 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.473098 1440 raft_consensus.cc:1226] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 7. Current term is 8. Ops: []
I20260808 01:11:14.477180 4378 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:11:14.501566 4598 consensus_meta.cc:287] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.638s user 0.014s sys 0.002s
W20260808 01:11:14.474879 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 142690us
I20260808 01:11:14.479408 1597 raft_consensus.cc:2396] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260808 01:11:14.455298 4269 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:14.481719 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:14.525360 1600 raft_consensus.cc:3037] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:14.525233 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.484366 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.525239 2205 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:48980
I20260808 01:11:14.490645 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:14.502345 1567 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:14.526678 2205 raft_consensus.cc:595] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:14.527151 2205 raft_consensus.cc:603] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:11:14.511093 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.527678 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.528635 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.528854 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.529436 1518 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 10.
I20260808 01:11:14.529752 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.530236 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:14.481217 4386 raft_consensus.cc:729] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:14.473074 1439 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:14.489961 4412 raft_consensus.cc:2781] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:14.506932 2171 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:14.503321 4574 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:14.530443 1599 raft_consensus.cc:2379] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:14.503569 4596 raft_consensus.cc:3037] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:14.527493 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.406775 4625 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:14.534420 1594 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:14.522269 4378 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:14.536989 1603 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:14.531651 4269 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:11:14.559348 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 78994.9us
I20260808 01:11:14.526641 4598 raft_consensus.cc:2726] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:14.556661 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.552619 1596 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:14.556727 1600 raft_consensus.cc:2454] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:14.559664 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.548612 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.537051 4401 raft_consensus.cc:2726] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:11:14.564671 1525 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:14.567517 1516 raft_consensus.cc:1261] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20260808 01:11:14.539806 4461 raft_consensus.cc:2726] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:14.566427 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.564757 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.508847 4491 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.565864 1445 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:11:14.568087 1516 pending_rounds.cc:76] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5: Aborting all ops after (but not including) 4
I20260808 01:11:14.567416 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:14.561170 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 136072us
I20260808 01:11:14.569077 1518 raft_consensus.cc:1261] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260808 01:11:14.570675 1516 pending_rounds.cc:97] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 4.5
I20260808 01:11:14.556411 4412 raft_consensus.cc:686] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:14.577400 1568 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:14.579337 1439 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8"
dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
mode: GRACEFUL
new_leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
from {username='slave'} at 127.0.0.1:51830
I20260808 01:11:14.580010 1439 raft_consensus.cc:595] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Received request to transfer leadership to TS 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:14.580382 1439 raft_consensus.cc:603] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:11:14.581362 4098 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:14.584024 1520 raft_consensus.cc:1261] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:11:14.585206 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:14.589203 4637 log.cc:924] Time spent T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20260808 01:11:14.589331 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.589852 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.590567 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.574014 1516 raft_consensus.cc:2894] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Aborting config change with OpId 4.5: Aborted: Op aborted by new leader
I20260808 01:11:14.591501 1516 raft_consensus.cc:2866] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:14.565342 4591 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:14.596998 1522 raft_consensus.cc:1226] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:14.598232 4651 consensus_queue.cc:1046] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:14.598758 4098 raft_consensus.cc:2781] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:14.613343 4098 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:14.613775 4098 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:14.613663 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:14.614651 1440 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:14.614964 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260808 01:11:14.616148 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 130383us
I20260808 01:11:14.597008 4649 consensus_queue.cc:1046] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.020s
I20260808 01:11:14.617056 2171 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:14.617467 4645 raft_consensus.cc:3032] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:14.617780 4553 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:14.618187 2171 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:14.618490 4461 raft_consensus.cc:2781] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:14.600358 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:14.619318 4593 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:14.620224 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.621093 1599 raft_consensus.cc:2379] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 5: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:14.621348 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.621789 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:11:14.622238 1599 raft_consensus.cc:1226] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:14.622577 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:11:14.622686 1597 raft_consensus.cc:1226] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:14.624244 1597 raft_consensus.cc:1261] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:11:14.628401 4412 consensus_queue.cc:237] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:14.631686 4098 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:14.634131 4098 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:14.603449 1446 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:14.640523 4099 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:14.614066 4651 raft_consensus.cc:3032] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:11:14.617909 4645 raft_consensus.cc:729] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:14.619035 4461 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:14.554186 4574 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.573702 4642 raft_consensus.cc:2781] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:14.618456 4553 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:14.643750 2171 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:14.605590 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.606810 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.645949 1521 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:14.604234 1523 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:14.653028 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:14.654732 1442 raft_consensus.cc:2454] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:14.654242 1598 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:14.666813 4568 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.610777 4477 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:14.679057 1446 raft_consensus.cc:2379] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:14.564031 4386 consensus_queue.cc:260] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.681834 1439 raft_consensus.cc:1226] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:14.681952 1599 raft_consensus.cc:1226] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260808 01:11:14.648943 4651 raft_consensus.cc:729] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:14.688233 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.699164 1519 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 5: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:14.656104 4481 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:14.655107 4461 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:11:14.689385 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 64598.1us
I20260808 01:11:14.701049 1448 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 9.
W20260808 01:11:14.706048 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 141669us
I20260808 01:11:14.711056 1597 raft_consensus.cc:1226] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:14.712898 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:14.655321 4642 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:14.682191 4491 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:14.724007 1567 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:14.715859 1446 raft_consensus.cc:2379] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 8: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:14.685510 4544 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:14.729396 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 152074us
I20260808 01:11:14.667104 4480 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.698122 4635 raft_consensus.cc:2781] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:14.732664 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:14.732965 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.733347 1448 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:14.741767 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.743131 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.744048 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.744287 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.745198 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.745565 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.745954 1518 raft_consensus.cc:2379] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:14.746549 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.746744 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.747311 1525 raft_consensus.cc:2379] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:14.747355 1518 raft_consensus.cc:2454] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:14.748171 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.748230 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.749431 4545 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.751951 2171 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:14.752249 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.752861 1521 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
W20260808 01:11:14.739547 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 142063us
I20260808 01:11:14.756953 2172 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:14.755420 1448 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:14.749668 4269 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.765794 1594 raft_consensus.cc:1226] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:14.767136 4269 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:14.679221 4591 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.769531 1489 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:14.772465 1602 raft_consensus.cc:2379] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 6: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:14.723999 4642 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:14.778985 1519 raft_consensus.cc:1226] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:14.779466 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:14.780278 1522 raft_consensus.cc:2379] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 4: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:14.785281 1568 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:14.774407 4406 raft_consensus.cc:2781] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Leader election won for term 9
W20260808 01:11:14.799608 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 70170.5us
I20260808 01:11:14.796607 1568 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:14.779775 4544 raft_consensus.cc:2781] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:14.811121 4098 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:11:14.788511 1516 raft_consensus.cc:2454] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:11:14.837340 1568 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:14.840162 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:14.844908 4480 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:14.804617 4491 raft_consensus.cc:2726] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:14.804744 4412 raft_consensus.cc:2781] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election won for term 7
W20260808 01:11:14.857350 4655 log.cc:924] Time spent T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.075s user 0.002s sys 0.000s
W20260808 01:11:14.859354 4586 consensus_meta.cc:287] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.939s user 0.009s sys 0.007s
I20260808 01:11:14.859865 4586 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.862792 1438 raft_consensus.cc:1226] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:14.863494 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:14.864388 2201 raft_consensus.cc:3037] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:14.868906 4635 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:14.869443 4635 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:14.869653 1440 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
W20260808 01:11:14.872248 4629 consensus_meta.cc:287] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.618s user 0.004s sys 0.007s
I20260808 01:11:14.872676 4629 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:14.873698 1523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58280 (request call id 2028) took 1084 ms. Trace:
I20260808 01:11:14.874218 1523 rpcz_store.cc:276] 0808 01:11:13.788787 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:13.793853 (+ 5066us) service_pool.cc:227] Handling call
0808 01:11:14.873466 (+1079613us) spinlock_profiling.cc:243] Waited 620 ms on lock 0x7b540023d2d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:14.873588 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1426640256}
I20260808 01:11:14.874943 1596 raft_consensus.cc:1226] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:14.875361 1604 raft_consensus.cc:2379] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 4: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:14.877319 4591 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:14.880925 4593 raft_consensus.cc:2726] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:14.718822 4645 consensus_queue.cc:260] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.881106 2201 raft_consensus.cc:2454] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:14.882097 4568 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:14.780093 4558 raft_consensus.cc:2781] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:14.885717 1599 raft_consensus.cc:1226] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: []
W20260808 01:11:14.886749 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 51045.8us
I20260808 01:11:14.874817 4629 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:11:14.891656 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 204945us
I20260808 01:11:14.897023 1408 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
W20260808 01:11:14.897141 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 86519.5us
W20260808 01:11:14.898890 1410 leader_election.cc:334] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 1.603s (SENT)
W20260808 01:11:14.887531 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 85268.1us
W20260808 01:11:14.788280 4659 log.cc:924] Time spent T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.066s user 0.002s sys 0.000s
I20260808 01:11:14.904989 2172 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:14.904574 1410 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:14.909926 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:14.911476 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:14.931453 1603 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:14.883513 4586 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:14.912588 4558 raft_consensus.cc:686] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:14.912618 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:11:14.913595 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 157075us
I20260808 01:11:14.904695 4098 raft_consensus.cc:686] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 9 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:14.920272 4645 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:14.921269 1444 raft_consensus.cc:1226] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 7. Current term is 8. Ops: []
I20260808 01:11:14.942772 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:14.926705 4574 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:14.893163 4412 raft_consensus.cc:686] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:14.927964 4553 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.912662 2171 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:14.783108 4571 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:14.927223 4642 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.932610 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:14.901937 1567 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:14.906401 4649 raft_consensus.cc:2781] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:14.944006 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:14.924577 4545 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:14.943174 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:14.969934 1598 raft_consensus.cc:2379] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 8: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:14.970055 1448 raft_consensus.cc:2379] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 9: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:14.927253 4295 raft_consensus.cc:2781] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:14.949957 1602 raft_consensus.cc:1226] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:14.950418 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:14.973935 1596 raft_consensus.cc:2379] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:14.843194 4651 consensus_queue.cc:260] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:14.943974 4593 raft_consensus.cc:2781] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:14.980913 1490 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:14.972318 1601 raft_consensus.cc:2354] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 11. Current term is 12.
W20260808 01:11:14.975899 1565 leader_election.cc:334] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.527s (SENT)
I20260808 01:11:14.981962 1565 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:14.977969 1517 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:14.942405 4480 consensus_queue.cc:1035] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260808 01:11:14.969772 4417 raft_consensus.cc:2726] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260808 01:11:14.991570 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 78998.7us
I20260808 01:11:14.944828 4461 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:14.993969 1490 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:14.996088 4544 raft_consensus.cc:686] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
W20260808 01:11:14.997051 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 104111us
I20260808 01:11:14.999639 4412 consensus_queue.cc:237] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:15.000226 4629 raft_consensus.cc:2726] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:11:15.000895 4597 raft_consensus.cc:2726] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:11:15.003160 4593 raft_consensus.cc:686] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:15.004151 4593 consensus_queue.cc:237] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.002612 4544 consensus_queue.cc:237] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.008066 4491 raft_consensus.cc:2726] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:11:15.010681 1490 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:15.011478 4597 raft_consensus.cc:2781] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:15.011961 4597 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:15.012351 4597 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:15.012709 4571 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 12 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:15.012446 4635 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.014523 4635 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:15.015307 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:15.017584 1448 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:11:14.991089 4649 raft_consensus.cc:686] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:15.016705 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:15.020805 4649 consensus_queue.cc:237] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:14.988261 4636 raft_consensus.cc:2781] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:15.028165 1448 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 12.
I20260808 01:11:15.028240 4098 consensus_queue.cc:237] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.034755 4642 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:15.036278 1568 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:11:14.993940 4654 consensus_queue.cc:1159] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" responder_term: 8 status { last_received { term: 7 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:11:15.039520 4480 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:15.040094 4480 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.042918 4480 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:15.044510 2205 raft_consensus.cc:1226] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 7. Current term is 8. Ops: []
I20260808 01:11:15.045292 4386 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:15.046002 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:15.046687 1448 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:15.028112 4558 consensus_queue.cc:237] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.028074 4597 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:15.052181 4597 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:15.053300 4461 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:14.992663 4652 raft_consensus.cc:2726] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:15.043541 4480 raft_consensus.cc:2781] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:11:14.991101 4651 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:15.054867 4480 raft_consensus.cc:686] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [term 12 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:15.056021 4480 consensus_queue.cc:237] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:15.060091 2171 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:11:15.060786 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 65528.7us
I20260808 01:11:15.060899 4461 raft_consensus.cc:2781] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:15.061336 4461 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:15.061579 2171 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:15.062426 4635 raft_consensus.cc:2726] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:15.063189 2171 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
W20260808 01:11:15.064215 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 68588.4us
I20260808 01:11:15.064939 1489 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:15.036424 4553 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:15.061746 4461 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:15.069717 2171 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:15.088270 4406 raft_consensus.cc:686] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 9 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:15.064019 4635 raft_consensus.cc:2781] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:15.156098 4406 consensus_queue.cc:237] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.157356 4635 raft_consensus.cc:686] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:15.158457 4635 consensus_queue.cc:237] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.051519 4378 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.164402 4378 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:15.099582 4553 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:11:15.172255 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 107360us
I20260808 01:11:15.173105 1489 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:15.174013 4477 raft_consensus.cc:2781] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:15.174444 4477 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:15.174799 4477 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:15.180344 2209 raft_consensus.cc:1226] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 6. Current term is 7. Ops: []
W20260808 01:11:15.186868 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 117210us
I20260808 01:11:15.043496 4596 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:15.198745 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:15.202611 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:15.203576 2204 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:15.205226 2201 raft_consensus.cc:1226] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:15.205351 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:15.206791 2201 raft_consensus.cc:1226] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:15.206720 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:15.208076 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:15.208271 2205 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:15.209372 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:15.217069 1489 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:15.229352 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:15.231185 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:15.232273 2206 raft_consensus.cc:2354] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4.
I20260808 01:11:15.233317 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:15.231278 1600 raft_consensus.cc:2354] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4.
I20260808 01:11:15.237573 2205 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
W20260808 01:11:15.244969 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 245763us
I20260808 01:11:15.251864 2209 raft_consensus.cc:1226] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEARNER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
W20260808 01:11:15.254626 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 352793us
W20260808 01:11:15.258886 1410 leader_election.cc:334] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 1.829s (SENT)
I20260808 01:11:15.259335 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:15.259553 1410 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:15.263551 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:15.264554 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:15.264798 2202 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:15.265427 2204 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:15.216814 4378 raft_consensus.cc:2781] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election won for term 6
W20260808 01:11:15.267280 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 50327us
I20260808 01:11:15.271301 1489 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:15.273893 1489 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:15.269222 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 298171us
I20260808 01:11:15.244797 4597 raft_consensus.cc:2726] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:15.279296 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:15.287940 3772 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:15.288652 3772 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.290710 1567 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:15.293685 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:15.294623 3772 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:15.297717 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:15.273237 4586 raft_consensus.cc:2726] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:11:15.226260 4417 consensus_queue.cc:1046] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260808 01:11:15.276072 4491 raft_consensus.cc:2726] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:15.262804 4295 raft_consensus.cc:686] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:15.216404 4664 consensus_queue.cc:1035] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20260808 01:11:15.307118 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:15.308192 2201 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:15.286710 4642 raft_consensus.cc:2726] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:15.286928 4378 raft_consensus.cc:686] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
W20260808 01:11:15.317663 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 43959.9us
I20260808 01:11:15.350888 1489 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:15.331660 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:15.315454 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 68409us
I20260808 01:11:15.357869 2204 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:11:15.373562 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:15.305369 4645 raft_consensus.cc:2741] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260808 01:11:15.371774 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 81148.8us
I20260808 01:11:15.373595 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:15.366818 2207 raft_consensus.cc:1226] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:15.380239 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:15.385126 1446 raft_consensus.cc:3037] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:15.325645 4491 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:15.377553 2201 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:11:15.390005 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:15.395943 1599 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 5: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
W20260808 01:11:15.396960 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 46217.8us
I20260808 01:11:15.398434 1489 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:15.395962 1410 leader_election.cc:334] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 1.872s (SENT)
I20260808 01:11:15.404210 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:15.405802 1410 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:15.407334 3772 raft_consensus.cc:2726] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:15.409963 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:15.410686 1596 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:15.378584 4642 raft_consensus.cc:2726] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:11:15.380537 4636 raft_consensus.cc:686] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:15.412209 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:15.412714 1487 leader_election.cc:334] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.901s (SENT)
I20260808 01:11:15.404372 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:15.420609 1444 raft_consensus.cc:1261] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:11:15.421386 1602 raft_consensus.cc:2379] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 6: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
W20260808 01:11:15.424500 4666 log.cc:924] Time spent T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.123s user 0.000s sys 0.001s
I20260808 01:11:15.423898 4378 consensus_queue.cc:237] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.431300 1487 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:15.420429 1597 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:11:15.401935 4386 consensus_queue.cc:1046] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260808 01:11:15.436134 4386 raft_consensus.cc:3032] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:15.436681 4386 raft_consensus.cc:729] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:15.437737 4386 consensus_queue.cc:260] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.439540 4386 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:15.442865 1446 raft_consensus.cc:2454] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:15.409694 4574 raft_consensus.cc:2726] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:15.444481 4574 consensus_queue.cc:1046] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:11:15.446249 1568 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:15.428906 4667 consensus_queue.cc:1035] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:11:15.453868 2202 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:15.429960 4596 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:15.470387 4636 consensus_queue.cc:237] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.409601 4491 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.450212 4553 raft_consensus.cc:2726] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:15.451917 4645 raft_consensus.cc:2781] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:15.481217 4645 raft_consensus.cc:686] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:15.482496 4645 consensus_queue.cc:237] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:15.466604 4417 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:15.484989 4417 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.487299 4417 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:15.488108 4553 raft_consensus.cc:2726] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:15.488495 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:15.491652 4491 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:15.466522 4574 consensus_queue.cc:1225] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer a4862d0938c84fc4bd840e4c17acae78 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260808 01:11:15.495669 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 119703us
I20260808 01:11:15.497618 4416 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:15.498219 4416 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.478185 1567 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:15.480962 4642 consensus_queue.cc:1046] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:11:15.480053 4654 consensus_queue.cc:1046] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:15.507808 1596 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 7.
I20260808 01:11:15.431456 4649 consensus_queue.cc:1035] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
W20260808 01:11:15.523108 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 91885.7us
I20260808 01:11:15.524143 1487 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:15.525032 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 108015us
I20260808 01:11:15.537755 1567 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:15.450980 4295 consensus_queue.cc:237] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.544370 4629 raft_consensus.cc:2781] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:15.550298 4568 raft_consensus.cc:2781] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:15.550983 4568 raft_consensus.cc:686] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:15.552006 4568 consensus_queue.cc:237] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.554488 4629 raft_consensus.cc:686] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:15.530040 4416 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:15.574643 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:15.577106 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:15.575542 1596 raft_consensus.cc:2362] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 8. Re-sending same reply.
W20260808 01:11:15.583300 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 181139us
W20260808 01:11:15.584436 2171 connection.cc:232] RPC call timeout handler was delayed by -1.3872604389907792s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2231s for a response
I20260808 01:11:15.593063 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:15.593796 1602 raft_consensus.cc:2362] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3. Re-sending same reply.
I20260808 01:11:15.594524 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:15.594969 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:15.595309 1603 raft_consensus.cc:2362] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6. Re-sending same reply.
I20260808 01:11:15.596524 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:15.597159 1594 raft_consensus.cc:1261] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:11:15.597754 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:15.598486 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:15.599619 1604 raft_consensus.cc:1226] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 7. Current term is 8. Ops: [5.8-7.9]
I20260808 01:11:15.600147 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:15.601047 1601 raft_consensus.cc:2362] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3. Re-sending same reply.
I20260808 01:11:15.602324 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:15.602926 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:15.605008 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:15.606244 1599 raft_consensus.cc:2396] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260808 01:11:15.610169 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:15.611055 1600 raft_consensus.cc:2362] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4. Re-sending same reply.
I20260808 01:11:15.539301 4517 consensus_queue.cc:1046] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260808 01:11:15.545173 4596 consensus_queue.cc:1035] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260808 01:11:15.613137 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:15.624114 2171 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:15.614645 4477 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.628098 4477 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:15.617136 4544 consensus_queue.cc:1035] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260808 01:11:15.639058 1602 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 8.
I20260808 01:11:15.634104 4629 consensus_queue.cc:237] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:15.647688 4461 consensus_meta.cc:287] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.551s user 0.011s sys 0.000s
I20260808 01:11:15.654245 4461 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.656107 2203 raft_consensus.cc:1226] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
W20260808 01:11:15.664378 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 45399.5us
I20260808 01:11:15.666299 2171 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:15.665911 4544 raft_consensus.cc:2726] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:15.667932 4544 raft_consensus.cc:3037] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:15.670114 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:15.670974 2201 raft_consensus.cc:2362] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6. Re-sending same reply.
I20260808 01:11:15.669598 4553 raft_consensus.cc:3032] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:15.672344 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:15.674057 2201 raft_consensus.cc:1261] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:11:15.674407 2204 raft_consensus.cc:1226] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:15.676611 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:15.677460 2207 raft_consensus.cc:1226] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:15.678355 2202 raft_consensus.cc:2396] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260808 01:11:15.679226 2208 raft_consensus.cc:2379] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 9: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:15.679987 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:15.672315 4553 raft_consensus.cc:729] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:15.681260 4553 consensus_queue.cc:260] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:15.682443 4553 raft_consensus.cc:3037] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:15.684175 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:15.685016 2206 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:15.685286 2200 raft_consensus.cc:1226] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:15.691936 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:15.690546 4461 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:15.771168 4568 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:15.771921 4568 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.775130 4568 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:15.771575 4544 raft_consensus.cc:2726] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:15.785089 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
W20260808 01:11:15.819317 1565 leader_election.cc:334] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571): Timed out: RequestConsensusVote RPC to 127.31.212.68:34571 timed out after 1.995s (SENT)
I20260808 01:11:15.828322 1565 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:15.831692 4568 raft_consensus.cc:2726] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:11:15.833791 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:15.835824 2204 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:15.842787 2200 raft_consensus.cc:1261] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260808 01:11:15.843228 2200 pending_rounds.cc:76] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd: Aborting all ops after (but not including) 8
I20260808 01:11:15.843482 2200 pending_rounds.cc:97] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260808 01:11:15.843724 2200 raft_consensus.cc:2866] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:15.848824 2200 raft_consensus.cc:1261] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
W20260808 01:11:15.848896 1338 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55756 (request call id 337) took 11334 ms (client timeout 15000 ms). Trace:
W20260808 01:11:15.850199 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 323597us
I20260808 01:11:15.854616 4689 consensus_queue.cc:1046] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:15.856779 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:15.851694 1489 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:15.850082 1338 rpcz_store.cc:269] 0808 01:11:04.514221 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:04.515489 (+ 1268us) service_pool.cc:227] Handling call
0808 01:11:15.848669 (+11333180us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 01:11:11.841394 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:11:12.005316 (+163922us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:11:12.018746 (+ 13430us) write_op.cc:435] Acquired schema lock
0808 01:11:12.018771 (+ 25us) tablet.cc:660] Decoding operations
0808 01:11:12.422325 (+403554us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:11:12.422504 (+ 179us) write_op.cc:641] Partition lock acquired for write op
0808 01:11:12.422566 (+ 62us) tablet.cc:683] Acquiring locks for 22 operations
0808 01:11:12.463173 (+ 40607us) tablet.cc:699] Row locks acquired
0808 01:11:12.463219 (+ 46us) write_op.cc:260] PREPARE: finished
0808 01:11:12.463522 (+ 303us) write_op.cc:270] Start()
0808 01:11:12.463752 (+ 230us) write_op.cc:276] Timestamp: P: 1786151472463443 usec, L: 0
0808 01:11:12.463811 (+ 59us) op_driver.cc:331] REPLICATION: starting
0808 01:11:12.553682 (+ 89871us) log.cc:842] Serialized 8867 byte log entry
0808 01:11:14.167122 (+1613440us) op_driver.cc:444] REPLICATION: finished
0808 01:11:14.599217 (+432095us) write_op.cc:301] APPLY: starting
0808 01:11:14.599349 (+ 132us) tablet.cc:1365] starting BulkCheckPresence
0808 01:11:14.599887 (+ 538us) tablet.cc:1368] finished BulkCheckPresence
0808 01:11:14.600018 (+ 131us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 01:11:14.602121 (+ 2103us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 01:11:14.602420 (+ 299us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=22
0808 01:11:14.602449 (+ 29us) write_op.cc:312] APPLY: finished
0808 01:11:14.602942 (+ 493us) log.cc:842] Serialized 152 byte log entry
0808 01:11:14.603112 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:11:14.603351 (+ 239us) write_op.cc:454] Released schema lock
0808 01:11:14.603422 (+ 71us) write_op.cc:341] FINISH: Updating metrics
0808 01:11:14.624407 (+ 20985us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:11:14.624438 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":510298,"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":317583,"apply.run_cpu_time_us":5166,"apply.run_wall_time_us":29618,"num_ops":22,"prepare.queue_time_us":475719,"prepare.run_cpu_time_us":67021,"prepare.run_wall_time_us":1547462,"raft.queue_time_us":195146,"raft.run_cpu_time_us":33648,"raft.run_wall_time_us":596771,"replication_time_us":1674964,"thread_start_us":1378804,"threads_started":4,"wal-append.queue_time_us":676602,"wal-append.run_cpu_time_us":27537,"wal-append.run_wall_time_us":2377982}]]}
I20260808 01:11:15.870353 4629 consensus_queue.cc:1035] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260808 01:11:15.870884 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 361529us
I20260808 01:11:15.872692 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:15.873376 2200 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:15.876004 4629 consensus_queue.cc:1046] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:11:15.877465 4568 raft_consensus.cc:3032] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:15.877888 4568 raft_consensus.cc:729] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:15.878996 4568 consensus_queue.cc:260] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.881695 1438 raft_consensus.cc:1261] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:11:15.882124 4568 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:15.880340 1448 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260808 01:11:15.884502 1448 raft_consensus.cc:1261] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:11:15.886468 1448 raft_consensus.cc:1261] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:11:15.891304 2200 raft_consensus.cc:1226] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:15.893232 4664 consensus_queue.cc:1035] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:11:15.893963 4544 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260808 01:11:15.894708 4664 consensus_queue.cc:1035] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 01:11:15.895303 4544 consensus_queue.cc:1035] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:11:15.899847 4293 consensus_queue.cc:1046] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 01:11:15.919631 4568 consensus_queue.cc:1046] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:15.920778 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:15.921785 1602 raft_consensus.cc:2379] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 5: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:15.928712 1440 raft_consensus.cc:1261] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260808 01:11:15.941947 1440 raft_consensus.cc:1261] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 12 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260808 01:11:15.943327 4517 consensus_queue.cc:1046] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:15.944581 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:15.948622 4654 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:11:15.950664 1565 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:15.951968 4629 raft_consensus.cc:2726] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:11:15.958393 2171 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:15.959492 4678 raft_consensus.cc:2726] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:11:15.960698 2171 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:15.961400 4640 raft_consensus.cc:3032] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:11:15.961875 4640 raft_consensus.cc:729] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:15.962839 4640 consensus_queue.cc:260] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:15.964325 4640 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:15.964371 4678 raft_consensus.cc:2726] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:11:15.978626 1439 raft_consensus.cc:1261] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:11:15.978876 1447 raft_consensus.cc:1261] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260808 01:11:16.003031 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 478984us
I20260808 01:11:16.004076 1487 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:16.006301 1487 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:16.011566 1487 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:16.422613 4699 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:16.423868 4699 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:16.608048 4295 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:16.608909 4295 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:16.611673 4295 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:16.636699 1442 raft_consensus.cc:1261] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:11:16.638152 4699 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:16.679562 4714 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:16.680251 4714 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:16.682554 4714 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:16.683884 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:16.693488 4714 consensus_queue.cc:1035] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:11:16.711477 4491 raft_consensus.cc:3032] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:16.716502 4491 raft_consensus.cc:729] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:16.717736 4491 consensus_queue.cc:260] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:16.719318 4491 raft_consensus.cc:3037] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:11:16.724550 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 773994us
I20260808 01:11:16.725327 1565 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:15.965780 2171 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
W20260808 01:11:16.731819 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 766037us
I20260808 01:11:16.732777 2171 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
W20260808 01:11:16.743050 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 776669us
W20260808 01:11:16.750440 1602 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.796s user 0.006s sys 0.002s
W20260808 01:11:16.750975 1602 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.797s user 0.007s sys 0.002s
I20260808 01:11:16.770017 4651 raft_consensus.cc:2781] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:16.777684 4729 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:16.778508 4729 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:16.780892 4729 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:16.770569 4651 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:16.792809 1565 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:16.792073 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 63784.5us
I20260808 01:11:16.787992 4651 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:16.804472 4720 raft_consensus.cc:2781] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:16.805085 4720 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:16.805621 4720 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:16.808099 4728 consensus_queue.cc:1035] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260808 01:11:16.814702 4640 consensus_meta.cc:287] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.850s user 0.011s sys 0.013s
I20260808 01:11:16.818998 4571 consensus_queue.cc:1035] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:11:16.819942 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:11:16.845538 4568 raft_consensus.cc:2726] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:11:16.851913 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 94811.9us
I20260808 01:11:16.840693 4640 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:16.851400 4723 raft_consensus.cc:2781] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election won for term 7
W20260808 01:11:16.859222 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 105422us
W20260808 01:11:16.865350 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 70848.6us
I20260808 01:11:16.883505 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:16.855254 4728 consensus_queue.cc:1035] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260808 01:11:16.889057 1440 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:11:16.885126 1446 raft_consensus.cc:2454] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 2.
I20260808 01:11:16.859970 4568 consensus_queue.cc:1035] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260808 01:11:16.863722 4640 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:16.909732 4667 raft_consensus.cc:3032] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:16.910246 4736 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:16.910324 4667 raft_consensus.cc:729] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:16.911059 4736 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:16.897624 4723 raft_consensus.cc:686] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:16.892781 4731 consensus_queue.cc:1035] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
W20260808 01:11:16.913575 1410 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.404s user 0.024s sys 0.063s
I20260808 01:11:16.914736 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:16.919348 1442 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:16.922755 4720 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:16.884189 4651 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:16.925751 4720 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:16.926447 4651 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:16.929862 2172 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:16.930706 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:16.933495 4640 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:16.933974 4740 raft_consensus.cc:2781] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:16.934414 4740 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:16.934736 4740 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:16.940248 4731 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:16.915573 4723 consensus_queue.cc:237] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:16.942173 4728 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:16.942305 4671 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:16.942785 4728 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:16.945708 4568 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:16.946331 4629 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:16.946271 4568 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:16.946916 4629 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:16.942919 4671 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:16.947409 1410 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.438s user 0.024s sys 0.063s
W20260808 01:11:16.948812 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 1.43935e+06us
I20260808 01:11:16.949601 1410 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:16.950507 1410 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:16.940884 4731 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:16.959000 1410 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:16.961187 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:16.961120 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:16.962495 2207 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEARNER]: Advancing to term 8
I20260808 01:11:16.962072 4293 consensus_queue.cc:1046] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.002s
I20260808 01:11:16.945763 4728 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:16.913363 4736 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:16.949234 4651 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:16.953650 4649 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:16.957245 4640 raft_consensus.cc:2781] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:16.957664 4590 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:16.970631 4586 raft_consensus.cc:2781] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:16.975971 4295 raft_consensus.cc:3032] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:11:16.988296 4295 raft_consensus.cc:729] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:16.989141 4295 consensus_queue.cc:260] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:16.990559 4295 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:16.992332 4593 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:16.993048 4593 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:16.993753 4671 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:11:16.996778 1547 debug-util.cc:398] Leaking SignalData structure 0x7b0800148b00 after lost signal to thread 32596
I20260808 01:11:16.997802 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:16.998095 1410 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:16.992270 2207 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [term 8 LEARNER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 8.
I20260808 01:11:17.000145 4593 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:16.980960 4568 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:17.004562 4401 raft_consensus.cc:3032] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:17.004949 4401 raft_consensus.cc:729] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:17.005498 4731 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:17.005856 4401 consensus_queue.cc:260] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.006915 4401 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:17.004189 2207 raft_consensus.cc:1226] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260808 01:11:17.010449 1410 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:17.015653 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 119273us
I20260808 01:11:16.911772 4667 consensus_queue.cc:260] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:16.975431 4640 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:17.025739 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:17.024580 1568 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:16.952217 4740 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:16.982512 4586 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:17.011121 4731 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:17.031998 4728 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260808 01:11:17.036458 4586 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:16.976629 4649 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:16.977766 4651 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:17.075593 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:17.021456 4293 raft_consensus.cc:2781] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:17.074918 1600 raft_consensus.cc:1261] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:11:17.026379 4640 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:17.042904 1602 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260808 01:11:17.027549 4568 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:17.026247 4269 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:11:17.078687 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 47784.7us
I20260808 01:11:17.032388 4667 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:17.024734 4699 raft_consensus.cc:2781] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:11:17.083681 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:17.084106 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:17.085681 2206 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:11:17.080402 4293 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
W20260808 01:11:17.092736 4742 log.cc:924] Time spent T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
I20260808 01:11:17.093379 4651 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:17.040243 4731 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.079500 4629 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:17.095400 4740 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:17.082181 4684 log.cc:924] Time spent T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260808 01:11:17.097391 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:17.097779 4568 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.101315 1438 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
I20260808 01:11:17.199821 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:17.102077 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:17.088251 4590 raft_consensus.cc:2726] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:17.112222 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:17.203925 2207 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:17.204192 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:17.112457 2205 raft_consensus.cc:1261] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:11:17.118119 3772 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
W20260808 01:11:17.120716 1547 debug-util.cc:398] Leaking SignalData structure 0x7b08001f0da0 after lost signal to thread 1376
I20260808 01:11:17.024194 4593 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:17.165427 1596 raft_consensus.cc:1261] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260808 01:11:17.132349 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 1.28061e+06us
I20260808 01:11:17.183149 1597 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:17.124963 4586 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:17.199306 1600 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
I20260808 01:11:17.197059 2202 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 8.
I20260808 01:11:17.209681 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:17.038563 4596 raft_consensus.cc:2781] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:11:17.210427 1439 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:17.200604 1442 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:17.203440 1599 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:17.205487 3772 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:17.206578 1547 debug-util.cc:398] Leaking SignalData structure 0x7b080016dfa0 after lost signal to thread 1473
W20260808 01:11:17.212237 1547 debug-util.cc:398] Leaking SignalData structure 0x7b080016df80 after lost signal to thread 1474
I20260808 01:11:17.206447 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:17.186985 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:17.212977 1438 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:17.213161 1445 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:17.204959 1448 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 8.
W20260808 01:11:17.213459 1547 debug-util.cc:398] Leaking SignalData structure 0x7b0800171780 after lost signal to thread 1551
W20260808 01:11:17.213827 1547 debug-util.cc:398] Leaking SignalData structure 0x7b0800171760 after lost signal to thread 1553
I20260808 01:11:17.184360 4293 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:17.191201 4699 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:17.216996 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:17.218381 3772 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:17.219758 4477 raft_consensus.cc:2781] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:17.224226 1448 raft_consensus.cc:2454] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:17.222604 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:17.225591 1568 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:17.234153 1568 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:11:17.235255 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 104697us
I20260808 01:11:17.236251 1568 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:17.237699 1568 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:17.239871 1410 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:17.169693 4269 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:17.211583 4596 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:17.228591 4477 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:17.232391 4699 raft_consensus.cc:3037] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:17.205614 4629 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260808 01:11:17.207345 4736 consensus_queue.cc:1035] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.048s
I20260808 01:11:17.252034 4728 raft_consensus.cc:2781] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:17.250644 4480 raft_consensus.cc:2781] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:11:17.212198 4593 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.259892 4596 raft_consensus.cc:3037] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:17.263259 4477 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:17.274314 1442 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 7.
I20260808 01:11:17.266151 1408 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:17.276571 2200 raft_consensus.cc:1226] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:17.294574 3772 raft_consensus.cc:2781] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:17.294946 3772 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:17.265205 4295 raft_consensus.cc:2726] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:17.295012 1408 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:17.276443 4728 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:17.278330 4480 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:17.298956 2209 raft_consensus.cc:2396] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 01:11:17.295271 3772 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:17.308566 4735 raft_consensus.cc:2781] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:17.308401 4755 consensus_queue.cc:1035] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:11:17.309033 4735 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:17.309453 4735 raft_consensus.cc:3037] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:17.310748 4743 raft_consensus.cc:2781] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:17.311355 4743 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:17.311920 4738 consensus_queue.cc:1035] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.112s
I20260808 01:11:17.313284 4412 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:17.313572 4747 log.cc:924] Time spent T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260808 01:11:17.313793 4412 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:17.315227 4751 log.cc:924] Time spent T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260808 01:11:17.304952 4728 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:17.316772 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:17.318627 4752 log.cc:924] Time spent T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260808 01:11:17.328240 4645 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:17.328998 4645 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:17.334007 4586 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:17.334827 4745 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:17.335487 4745 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:17.342947 4649 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:17.345269 4731 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:17.303993 4480 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:17.350564 2203 raft_consensus.cc:1261] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
W20260808 01:11:17.363847 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 124096us
W20260808 01:11:17.370525 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 55688.1us
I20260808 01:11:17.372999 1410 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:17.377790 1410 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:17.387554 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:17.389261 3772 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.369748 4295 raft_consensus.cc:2781] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:11:17.412488 4295 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:17.412916 4760 raft_consensus.cc:2781] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:11:17.413127 4295 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:17.413479 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:17.414212 2203 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:11:17.414366 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 203502us
I20260808 01:11:17.417007 4729 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 01:11:17.418809 4733 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:11:17.410512 1443 raft_consensus.cc:2379] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 7: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:17.411010 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:17.419982 1448 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:17.406013 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:17.399708 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:17.421948 1444 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:17.423044 2172 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:17.413614 4760 raft_consensus.cc:686] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:17.425124 1568 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:17.311772 4743 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:17.428333 1597 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:17.406097 4412 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:17.430657 1445 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 8.
I20260808 01:11:17.431493 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:17.433552 3772 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:17.436998 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:17.437861 1568 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:11:17.440887 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 63196.3us
I20260808 01:11:17.441551 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:17.441617 1410 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:17.381283 4593 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:17.444226 1547 debug-util.cc:398] Leaking SignalData structure 0x7b0800153ec0 after lost signal to thread 1629
W20260808 01:11:17.444559 1547 debug-util.cc:398] Leaking SignalData structure 0x7b08003d1d20 after lost signal to thread 1630
I20260808 01:11:17.392236 4759 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:17.442427 1602 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
W20260808 01:11:17.445595 1547 debug-util.cc:398] Leaking SignalData structure 0x7b080039e3c0 after lost signal to thread 2233
I20260808 01:11:17.405337 4269 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:11:17.465821 1547 debug-util.cc:398] Leaking SignalData structure 0x7b0800339120 after lost signal to thread 2234
I20260808 01:11:17.397979 4745 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:17.473421 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:17.429100 4635 raft_consensus.cc:2781] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:17.448590 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:17.437107 4720 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:17.479182 2171 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:17.479807 4738 consensus_queue.cc:1035] T 5353a0d85aff48078011ca6efaf9ddfa P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:11:17.482739 4480 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:17.483153 4728 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:17.484673 4759 raft_consensus.cc:2726] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:17.484654 4735 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:17.486917 2203 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:17.443465 4629 raft_consensus.cc:2781] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:17.489406 1410 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:17.443607 4412 raft_consensus.cc:2781] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:17.423678 4545 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20260808 01:11:17.495268 1568 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:17.473204 4586 raft_consensus.cc:2726] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:17.449571 4568 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:17.487254 4269 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:17.509431 1445 raft_consensus.cc:1226] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:17.511940 1410 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:17.512171 4759 raft_consensus.cc:2741] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260808 01:11:17.512722 4759 raft_consensus.cc:2726] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:11:17.495677 4720 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.514051 4517 raft_consensus.cc:2781] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:11:17.517212 4728 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:17.517513 2201 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:17.517777 4517 raft_consensus.cc:686] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 8 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:17.519078 4517 consensus_queue.cc:237] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.502502 4629 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:17.520889 4629 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:17.525838 1410 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:17.531400 4763 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:17.531989 4763 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.533054 4735 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:17.534071 4763 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:17.502628 4412 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:17.542146 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:17.543076 1440 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:17.522612 3772 consensus_queue.cc:1035] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:11:17.553429 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:17.554150 1447 raft_consensus.cc:3037] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:17.544776 4671 raft_consensus.cc:1050] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:11:17.517264 4720 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:17.566843 1410 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:17.548408 4412 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:17.549657 4763 raft_consensus.cc:2726] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:17.580931 3772 raft_consensus.cc:2726] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:11:17.513700 4477 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.525164 4269 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:17.593657 1408 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:17.595082 3772 raft_consensus.cc:2781] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:17.597067 3772 raft_consensus.cc:686] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:17.598569 4295 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.599658 4269 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:17.499558 4760 consensus_queue.cc:237] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:17.598882 3772 consensus_queue.cc:237] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:17.673962 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 124383us
I20260808 01:11:17.603716 1440 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:17.604257 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:17.687677 1444 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 8.
W20260808 01:11:17.673836 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 71665.3us
I20260808 01:11:17.688683 1447 raft_consensus.cc:2454] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:17.619807 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:17.608839 1547 debug-util.cc:398] Leaking SignalData structure 0x7b08002569c0 after lost signal to thread 4574
I20260808 01:11:17.689740 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:17.634188 1442 raft_consensus.cc:1261] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:11:17.709499 2200 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:11:17.611640 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 132531us
I20260808 01:11:17.713500 2171 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:17.712009 1410 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:17.723783 1444 raft_consensus.cc:1261] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:11:17.615912 4678 consensus_queue.cc:1046] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.100s
I20260808 01:11:17.605386 4553 raft_consensus.cc:2726] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:17.612356 4295 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 9 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:17.610601 4591 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:17.614064 1339 catalog_manager.cc:5495] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 3 to 5, leader changed from 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 5 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:17.622648 4689 raft_consensus.cc:3032] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:17.613009 4723 raft_consensus.cc:2781] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:17.609658 4597 raft_consensus.cc:2726] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260808 01:11:17.717778 4667 consensus_meta.cc:287] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.526s user 0.010s sys 0.010s
I20260808 01:11:17.610989 4574 raft_consensus.cc:2726] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:17.743757 4461 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:17.765332 4640 consensus_meta.cc:287] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.631s user 0.006s sys 0.015s
I20260808 01:11:17.765019 4461 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.765753 4640 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:17.768617 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 132531us
W20260808 01:11:17.756002 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 44143.4us
I20260808 01:11:17.771433 1410 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:17.757252 1339 catalog_manager.cc:5495] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 3 to 5, leader changed from 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 5 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:17.759896 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:17.751097 4760 raft_consensus.cc:2726] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:17.762425 4664 consensus_queue.cc:1035] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 01:11:17.759258 4591 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:17.601846 4596 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:17.754050 4593 raft_consensus.cc:2781] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:17.613175 4635 raft_consensus.cc:686] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:17.756629 4480 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:17.611670 4629 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:17.787160 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:17.784271 1339 catalog_manager.cc:5495] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 6 to 7, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 7 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:11:17.773674 4699 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:17.787791 4723 raft_consensus.cc:686] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
W20260808 01:11:17.788724 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 1.77705e+06us
I20260808 01:11:17.789354 4491 raft_consensus.cc:2726] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:17.757663 4689 raft_consensus.cc:729] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
W20260808 01:11:17.777827 4293 consensus_meta.cc:287] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.540s user 0.005s sys 0.006s
I20260808 01:11:17.801338 1339 catalog_manager.cc:5495] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:11:17.609970 4671 consensus_queue.cc:237] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.804646 1487 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:17.778189 4640 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:17.782460 4759 raft_consensus.cc:2726] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:17.791096 4593 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:17.833592 1339 catalog_manager.cc:5495] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 3 to 5, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 5 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:17.799674 4642 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:17.830140 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 58810.4us
I20260808 01:11:17.844926 1410 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:17.845124 1487 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:17.847359 1339 catalog_manager.cc:5495] T d6b8d42642b84d3f9a2adbbaf3fd5417 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 7 to 10, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 10 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:11:17.849743 1339 catalog_manager.cc:5495] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 3 to 5, leader changed from 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 5 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:11:17.858382 1339 catalog_manager.cc:5495] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 6 to 7, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 7 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:11:17.862265 1339 catalog_manager.cc:5495] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 2 to 3, leader changed from 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 3 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:11:17.864540 1339 catalog_manager.cc:5495] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 2 to 5, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 5 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:11:17.835520 4597 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:17.830983 4574 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:17.873083 1487 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
W20260808 01:11:17.867661 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 659006us
I20260808 01:11:17.837073 4553 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:17.838111 4690 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:17.845770 4593 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:17.872545 4477 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:17.877877 1339 catalog_manager.cc:5495] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 3 to 4, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 4 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:11:17.891417 1442 raft_consensus.cc:1226] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:17.892695 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:17.885105 1339 catalog_manager.cc:5495] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 1 to 4, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 4 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:11:17.835172 4635 consensus_queue.cc:237] T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.897011 1339 catalog_manager.cc:5495] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 1 to 4, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 4 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:11:17.872102 4787 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:17.838322 4293 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.846249 4688 consensus_queue.cc:1046] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260808 01:11:17.902736 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:17.904106 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:17.916014 1518 raft_consensus.cc:3037] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Advancing to term 12
I20260808 01:11:17.916139 1520 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:17.905874 1525 raft_consensus.cc:1226] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260808 01:11:17.906613 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:17.903551 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:17.918533 1516 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:17.909300 1339 catalog_manager.cc:5495] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 2 to 4, leader changed from 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) to a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67). New cstate: current_term: 4 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:11:17.875819 4755 raft_consensus.cc:2726] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260808 01:11:17.910646 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 65795.1us
I20260808 01:11:17.910622 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:17.923463 1519 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:17.923511 1410 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:17.913226 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:17.913257 1487 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:17.855587 4642 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:17.920717 1339 catalog_manager.cc:5495] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 2 to 3, leader changed from 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 3 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:17.926275 1339 catalog_manager.cc:5495] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "a4862d0938c84fc4bd840e4c17acae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } }
I20260808 01:11:17.929317 4293 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:17.929848 1521 raft_consensus.cc:1226] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:17.930161 1443 raft_consensus.cc:1226] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:17.931500 4699 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:17.900063 4477 raft_consensus.cc:2781] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election won for term 4
W20260808 01:11:17.933152 2171 leader_election.cc:334] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 1.829s (SENT)
I20260808 01:11:17.933980 1517 raft_consensus.cc:1261] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
W20260808 01:11:17.927600 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 45877us
I20260808 01:11:17.935235 1517 raft_consensus.cc:1226] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260808 01:11:17.934784 1339 catalog_manager.cc:5495] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 reported cstate change: term changed from 2 to 4, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66). New cstate: current_term: 4 leader_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } }
I20260808 01:11:17.936858 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:17.938974 2200 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 9.
I20260808 01:11:17.940269 1408 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:17.941206 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:17.941848 1524 raft_consensus.cc:3037] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 4
I20260808 01:11:17.945003 4759 raft_consensus.cc:2781] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:17.945693 4596 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:17.884474 4690 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:17.945724 1328 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet a83a963773094d6db6495b7009fd8690 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6b09dbefd24b4d52b385c4dff5697dc5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:17.946408 4591 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:17.912238 4636 consensus_queue.cc:1046] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260808 01:11:17.945111 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:17.948180 1516 raft_consensus.cc:1261] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260808 01:11:17.948518 4636 raft_consensus.cc:3032] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:17.948748 4629 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:17.949060 4636 raft_consensus.cc:729] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:17.950480 1525 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:17.948580 1516 pending_rounds.cc:76] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5: Aborting all ops after (but not including) 4
I20260808 01:11:17.953589 1516 pending_rounds.cc:97] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260808 01:11:17.953837 1516 raft_consensus.cc:2866] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:17.879612 4574 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.884662 4553 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:17.886029 4597 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.968314 1518 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 12.
I20260808 01:11:17.975281 4642 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:17.975656 4597 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:17.975912 4553 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:17.976452 1489 leader_election.cc:398] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4.
I20260808 01:11:17.976262 4591 consensus_queue.cc:1046] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 01:11:17.976733 1489 leader_election.cc:401] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:11:17.976799 4545 raft_consensus.cc:3032] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:17.945689 4759 raft_consensus.cc:686] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:17.988209 4785 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:17.988523 4759 consensus_queue.cc:237] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:17.988780 4785 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:17.990813 4785 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:17.995410 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:11:17.980990 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 57537.2us
I20260808 01:11:17.996853 1410 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:17.948150 4690 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:17.997334 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:17.997947 1447 raft_consensus.cc:2379] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 3: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
W20260808 01:11:18.000495 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 87487.8us
I20260808 01:11:18.008294 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:18.009002 1518 raft_consensus.cc:1261] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260808 01:11:18.009080 1440 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 7
I20260808 01:11:18.009944 1524 raft_consensus.cc:2454] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:18.011436 1524 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:11:17.886482 4689 consensus_queue.cc:260] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.965269 4699 raft_consensus.cc:2781] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:11:18.022347 4750 raft_consensus.cc:2781] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:18.022915 4750 raft_consensus.cc:686] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:18.023245 4553 raft_consensus.cc:3032] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:18.023675 4553 raft_consensus.cc:729] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:17.902632 4723 consensus_queue.cc:237] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:17.982128 4545 raft_consensus.cc:729] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:18.024061 1440 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:17.982738 4688 raft_consensus.cc:3032] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:18.033515 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:18.035306 1518 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 8
I20260808 01:11:18.035594 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:18.035646 1410 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:17.943735 4743 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.042035 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:18.012265 4642 raft_consensus.cc:2781] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:18.040339 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:18.028429 4778 log.cc:924] Time spent T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.184s user 0.001s sys 0.000s
I20260808 01:11:18.042894 2209 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEARNER]: Advancing to term 7
I20260808 01:11:18.046911 1519 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:18.036365 1524 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:18.051260 4690 consensus_queue.cc:1046] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:11:18.023774 4750 consensus_queue.cc:237] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.024545 4553 consensus_queue.cc:260] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.047731 4733 consensus_queue.cc:1046] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.004s
I20260808 01:11:18.071585 4733 raft_consensus.cc:3032] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:18.072103 4733 raft_consensus.cc:729] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:18.072257 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:18.072952 4733 consensus_queue.cc:260] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.052277 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:18.053107 4591 raft_consensus.cc:2726] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4.
W20260808 01:11:18.054061 4783 log.cc:924] Time spent T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.159s user 0.000s sys 0.001s
W20260808 01:11:18.054883 4774 log.cc:924] Time spent T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.143s user 0.001s sys 0.000s
I20260808 01:11:18.054778 4491 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:18.055723 4771 log.cc:924] Time spent T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.215s user 0.001s sys 0.000s
I20260808 01:11:18.056216 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:18.034823 4699 raft_consensus.cc:686] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:18.020684 4755 raft_consensus.cc:2726] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 01:11:18.053220 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 81089.2us
I20260808 01:11:18.065806 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:18.066263 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:18.067085 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:18.119212 1442 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:18.065610 1525 raft_consensus.cc:1261] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260808 01:11:18.068264 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:18.121028 1448 raft_consensus.cc:2379] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 7: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:18.109794 4788 raft_consensus.cc:2726] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:18.069268 1516 raft_consensus.cc:1261] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:11:18.069727 1445 raft_consensus.cc:1226] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:18.069833 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:18.029840 4574 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:18.071512 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:18.075232 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:18.126922 1524 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:18.127190 1519 raft_consensus.cc:2396] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:11:18.052388 4461 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:18.037562 4295 raft_consensus.cc:2781] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:11:17.922416 4787 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.038622 4688 raft_consensus.cc:729] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:18.109886 1438 raft_consensus.cc:1226] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 2. Current term is 3. Ops: []
I20260808 01:11:18.081372 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:18.152565 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:18.153112 2206 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:18.079710 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:18.086416 1518 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 8.
I20260808 01:11:18.080868 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:18.155599 1487 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:17.977562 4636 consensus_queue.cc:260] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.074225 4733 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:18.076617 4642 raft_consensus.cc:686] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:18.159310 4642 consensus_queue.cc:237] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.160241 4699 consensus_queue.cc:237] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.062137 4645 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:18.150938 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:18.173945 1519 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:18.174629 2203 raft_consensus.cc:1226] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:18.155556 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:18.148437 4295 raft_consensus.cc:686] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:18.168839 1442 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:18.137265 4788 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:18.076615 4412 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:18.177487 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:18.186993 1518 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:18.175639 1524 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:18.187822 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:18.189358 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:18.190071 1519 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:18.179495 4574 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:18.132450 4491 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.190794 4574 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.179221 4597 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:18.192241 4597 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:18.192943 4491 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:18.194332 4597 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:18.196017 4690 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:18.196599 4690 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.199368 4690 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:18.204716 4743 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:18.205780 4688 consensus_queue.cc:260] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:18.115972 4545 consensus_queue.cc:260] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.209205 4636 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:18.208847 4295 consensus_queue.cc:237] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:18.211962 4791 raft_consensus.cc:2781] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:18.212545 4791 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:18.212960 4791 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:11:18.213739 4461 consensus_queue.cc:1159] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" responder_term: 4 status { last_received { term: 3 index: 5 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:11:18.216423 4788 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.219228 4788 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:18.221107 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:18.183099 4785 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:18.221732 2207 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 8.
I20260808 01:11:18.222309 4785 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:18.222955 1410 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:18.224490 4785 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:18.225365 4586 raft_consensus.cc:2781] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election won for term 5
W20260808 01:11:18.225838 4792 log.cc:924] Time spent T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260808 01:11:18.226044 4586 raft_consensus.cc:686] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:18.228688 4412 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:18.230212 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:18.231465 1487 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:18.224454 1410 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:18.234405 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:18.238112 2209 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEARNER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 7.
I20260808 01:11:18.235565 4477 raft_consensus.cc:686] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:18.239444 4763 raft_consensus.cc:2781] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:18.268245 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:18.247854 4763 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:18.269198 1517 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:18.270653 1410 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:18.184017 4591 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:18.285629 3772 raft_consensus.cc:2741] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260808 01:11:18.227371 4586 consensus_queue.cc:237] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.275421 4763 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:18.286484 4788 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:11:18.386652 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:18.388012 1518 raft_consensus.cc:2396] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for term 8 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
W20260808 01:11:18.389495 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 556411us
I20260808 01:11:18.390576 1565 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:18.392812 4654 raft_consensus.cc:2726] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:18.393697 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:18.394001 1565 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:18.396802 1517 raft_consensus.cc:2354] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6.
I20260808 01:11:18.393554 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:18.397675 1521 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:18.399533 1410 leader_election.cc:398] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Vote denied by peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093) with higher term. Message: Invalid argument: T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6.
I20260808 01:11:18.399820 1410 leader_election.cc:401] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20260808 01:11:18.353251 4774 log.cc:924] Time spent T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260808 01:11:18.402699 1517 raft_consensus.cc:1261] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260808 01:11:18.404800 3772 raft_consensus.cc:2726] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093) with higher term. Message: Invalid argument: T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6.
I20260808 01:11:18.406054 1517 pending_rounds.cc:76] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5: Aborting all ops after (but not including) 5
I20260808 01:11:18.406361 1517 pending_rounds.cc:97] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260808 01:11:18.406684 1517 raft_consensus.cc:2866] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:18.407699 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:18.415200 3772 consensus_queue.cc:1046] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:11:18.418776 3772 consensus_queue.cc:1225] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer 95b661b03f1d460b9657e61d4b0247cd log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260808 01:11:18.419466 3772 raft_consensus.cc:3032] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:18.419986 3772 raft_consensus.cc:729] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:18.395704 4788 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:18.427847 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:18.428723 1517 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:18.429567 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:18.428841 3772 consensus_queue.cc:260] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.436849 3772 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:18.438287 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:18.341601 4591 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.447403 1597 raft_consensus.cc:1226] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 6. Current term is 7. Ops: [6.9-6.9]
I20260808 01:11:18.408082 4729 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:18.450014 1568 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:18.453253 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:18.458577 1598 raft_consensus.cc:1261] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:11:18.462373 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:18.454186 1596 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:18.430049 4788 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:11:18.471392 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 77478.8us
I20260808 01:11:18.482995 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:18.483685 1517 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:18.484611 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:18.485399 1518 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:18.486440 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:11:18.486857 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 66889.7us
I20260808 01:11:18.487032 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:18.487912 1603 raft_consensus.cc:2379] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 6: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:18.487320 1602 raft_consensus.cc:2354] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 6. Current term is 7.
I20260808 01:11:18.489667 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:18.490630 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:18.491371 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:18.492553 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:18.422660 4477 consensus_queue.cc:237] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.494856 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:18.495249 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:18.495600 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:18.486351 4793 log.cc:924] Time spent T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20260808 01:11:18.529273 1603 raft_consensus.cc:2379] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 5: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:18.534029 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:18.534834 1603 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:18.492051 1602 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:18.536175 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:18.536118 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:18.538064 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:18.537060 1602 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:18.539167 1603 raft_consensus.cc:2379] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 3: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:18.548739 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:18.466890 4640 consensus_queue.cc:1035] T a19c1759594d4722abadf576352f9a98 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260808 01:11:18.498842 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:18.552016 1601 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:18.532294 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:18.487530 4568 raft_consensus.cc:2726] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:18.528622 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:18.486753 4295 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:18.500734 1565 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:18.459954 4729 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:18.552302 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:18.554385 1603 raft_consensus.cc:1226] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260808 01:11:18.549782 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:11:18.557403 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 107521us
W20260808 01:11:18.592213 4801 log.cc:924] Time spent T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
W20260808 01:11:18.599292 2171 leader_election.cc:334] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 1.620s (SENT)
I20260808 01:11:18.552379 4629 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:18.593075 4795 log.cc:924] Time spent T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.248s user 0.001s sys 0.000s
I20260808 01:11:18.493753 4574 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:18.493650 4787 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:18.606488 1518 raft_consensus.cc:1261] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20260808 01:11:18.605558 4593 consensus_meta.cc:287] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.708s user 0.005s sys 0.007s
I20260808 01:11:18.609035 1568 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:18.609156 1601 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
W20260808 01:11:18.606629 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 106007us
I20260808 01:11:18.610651 1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 620) took 1163 ms. Trace:
I20260808 01:11:18.609575 1602 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 9.
I20260808 01:11:18.610761 2171 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:18.625959 1598 raft_consensus.cc:2379] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 5: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
W20260808 01:11:18.608765 1490 connection.cc:232] RPC call timeout handler was delayed by -1.2904100129962899s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19440000000000002s for a response
W20260808 01:11:18.626682 1490 leader_election.cc:334] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.680s (SENT)
I20260808 01:11:18.627193 1490 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:18.628079 1597 raft_consensus.cc:1261] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:11:18.629526 1568 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:18.605062 4412 raft_consensus.cc:2781] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:18.633531 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:18.625500 1599 rpcz_store.cc:276] 0808 01:11:17.440847 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:17.441003 (+ 156us) service_pool.cc:227] Handling call
0808 01:11:17.465880 (+ 24877us) raft_consensus.cc:1457] Updating replica for 3 ops
0808 01:11:17.498963 (+ 33083us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:17.498996 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0808 01:11:17.502889 (+ 3893us) log.cc:842] Serialized 78 byte log entry
0808 01:11:17.612515 (+109626us) spinlock_profiling.cc:243] Waited 3.02 ms on lock 0x7b18002f5a80. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349af9345 00007fe352930a39 00007fe349b168bb 00007fe349b14cbf 00007fe349b16e22 00007fe34c6c05bb 00007fe34c6c6063 00007fe34c6c6411 00007fe34ca135cf
0808 01:11:17.758748 (+146233us) raft_consensus.cc:1631] Marking committed up to 10
0808 01:11:17.767455 (+ 8707us) log.cc:842] Serialized 14 byte log entry
0808 01:11:17.768348 (+ 893us) log.cc:842] Serialized 14 byte log entry
0808 01:11:17.781775 (+ 13427us) log.cc:842] Serialized 14 byte log entry
0808 01:11:17.985615 (+203840us) spinlock_profiling.cc:243] Waited 1.68 ms on lock 0x7b18002f5a80. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349b17764 00007fe349b1ab11 00007fe349b1aac9 00007fe349b1aa59 00007fe349b1aa21 00007fe349b19d1d 00007fe352d7b204 00007fe352d7b039 00007fe349af9dc6
0808 01:11:18.345131 (+359516us) spinlock_profiling.cc:243] Waited 8.74 ms on lock 0x7b18002f5a80. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349a8e4ca 00007fe35316ff99 00007fe3508e7bcf 00007fe349b177b6 00007fe349b1ab11 00007fe349b1aac9 00007fe349b1aa59 00007fe349b1aa21 00007fe349b19d1d
0808 01:11:18.385203 (+ 40072us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:18.385288 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:18.558151 (+172863us) raft_consensus.cc:1671] finished
0808 01:11:18.558172 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:18.603786 (+ 45614us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1176398,"prepare.queue_time_us":16192,"prepare.run_cpu_time_us":158,"prepare.run_wall_time_us":2808,"raft.queue_time_us":656803,"raft.run_cpu_time_us":536,"raft.run_wall_time_us":11617,"spinlock_wait_cycles":30932480,"thread_start_us":146041,"threads_started":1,"wal-append.queue_time_us":327783}
I20260808 01:11:18.636039 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:11:18.613903 4796 log.cc:924] Time spent T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.324s user 0.001s sys 0.000s
I20260808 01:11:18.637682 1520 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:18.645272 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:18.646010 1520 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:18.632455 4593 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.643106 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:18.649400 1568 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:18.649942 1518 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:18.650307 2203 raft_consensus.cc:1226] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:18.612422 4480 raft_consensus.cc:2781] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 01:11:18.639215 4799 log.cc:924] Time spent T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.153s user 0.001s sys 0.000s
W20260808 01:11:18.676707 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 50048.3us
W20260808 01:11:18.677754 1490 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.727s (SENT) [suppressed 53 similar messages]
I20260808 01:11:18.649076 4649 raft_consensus.cc:2781] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:11:18.682536 1518 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 9.
I20260808 01:11:18.686369 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:18.687275 1518 raft_consensus.cc:2379] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 7: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:18.688174 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:18.694643 4791 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.695927 4728 consensus_queue.cc:1035] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:11:18.697921 4729 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:11:18.698655 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 49338.1us
I20260808 01:11:18.699301 4671 raft_consensus.cc:2781] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:18.638095 4574 raft_consensus.cc:2726] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:11:18.663183 4480 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:18.610339 4295 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.682063 4654 raft_consensus.cc:2781] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:18.702669 4649 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:18.711934 4678 raft_consensus.cc:2726] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:11:18.713995 4636 consensus_meta.cc:287] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.504s user 0.011s sys 0.000s
I20260808 01:11:18.661206 4412 raft_consensus.cc:686] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:18.616142 4629 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.715288 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:18.819772 1521 raft_consensus.cc:2362] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6. Re-sending same reply.
I20260808 01:11:18.734164 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:18.734550 1520 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:18.735095 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:18.741441 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:18.742061 1445 raft_consensus.cc:1261] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:11:18.747977 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:18.748469 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:18.747211 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:18.779681 1603 raft_consensus.cc:1226] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 2. Current term is 3. Ops: []
W20260808 01:11:18.718678 4733 consensus_meta.cc:287] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.546s user 0.007s sys 0.005s
I20260808 01:11:18.783162 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:18.776506 4401 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:18.827431 1446 raft_consensus.cc:2379] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:18.827756 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:18.829082 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 151363us
I20260808 01:11:18.832278 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:18.833096 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:18.833644 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:18.835249 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:18.835433 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:18.836830 1490 connection.cc:232] RPC call timeout handler was delayed by -1.0831130990118254s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26680000000000004s for a response
I20260808 01:11:18.839556 1603 raft_consensus.cc:1226] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:18.836225 1602 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:18.835808 1521 raft_consensus.cc:2362] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 8. Re-sending same reply.
I20260808 01:11:18.746810 4480 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:18.741755 4591 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:18.838526 1490 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:18.846046 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:18.781256 4671 raft_consensus.cc:686] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:18.851642 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:18.853791 4591 raft_consensus.cc:2726] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:18.854066 4412 consensus_queue.cc:237] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:18.857688 4629 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:18.846421 4401 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.860056 1490 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:18.860952 4401 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:18.861122 4591 consensus_queue.cc:1046] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:11:18.862659 4591 raft_consensus.cc:2726] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:18.863423 4591 raft_consensus.cc:3032] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:18.864091 4591 raft_consensus.cc:729] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:18.865264 4591 consensus_queue.cc:260] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:18.866617 4295 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:18.867069 4591 raft_consensus.cc:3037] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:18.871272 4597 consensus_queue.cc:1046] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:11:18.872742 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:18.873068 4642 raft_consensus.cc:3032] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:11:18.873538 1521 raft_consensus.cc:2379] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 7: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:18.873659 4642 raft_consensus.cc:729] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:18.874641 4763 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.874809 4642 consensus_queue.cc:260] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:18.874837 1521 raft_consensus.cc:1261] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:11:18.878216 1490 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:18.827750 4654 raft_consensus.cc:686] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:18.836644 4649 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:18.848289 4593 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:18.867424 4788 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:18.930773 1522 raft_consensus.cc:1226] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:18.931169 1521 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "a4862d0938c84fc4bd840e4c17acae78"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:11:18.932003 1521 raft_consensus.cc:595] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Received request to transfer leadership to TS a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:18.932446 1521 raft_consensus.cc:603] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:11:18.884281 4713 raft_consensus.cc:2726] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260808 01:11:18.936071 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 102846us
W20260808 01:11:18.927489 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 328212us
I20260808 01:11:18.934830 1516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2083) took 1007 ms. Trace:
I20260808 01:11:18.917411 4805 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:18.945326 1516 rpcz_store.cc:276] 0808 01:11:17.927569 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:17.993509 (+ 65940us) service_pool.cc:227] Handling call
0808 01:11:18.064800 (+ 71291us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:18.121945 (+ 57145us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:18.924639 (+802694us) spinlock_profiling.cc:243] Waited 47.1 ms on lock 0x7b54000a64d0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:18.934756 (+ 10117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":108431744}
I20260808 01:11:18.876117 4671 consensus_queue.cc:237] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:11:18.960968 2171 leader_election.cc:334] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 1.677s (SENT)
I20260808 01:11:18.905705 4629 consensus_queue.cc:1035] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260808 01:11:19.012001 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:19.014349 1597 raft_consensus.cc:1261] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:11:19.032107 1446 raft_consensus.cc:1261] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:11:19.035450 1521 raft_consensus.cc:1226] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:19.037498 4788 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 10 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:19.037788 1521 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:19.031971 4401 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:19.040243 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:19.041553 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:19.053876 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:19.042160 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:19.047228 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:19.055610 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:19.048919 2209 raft_consensus.cc:1261] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260808 01:11:19.013087 4713 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:19.078055 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:19.062026 1446 raft_consensus.cc:2396] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
W20260808 01:11:19.064248 1517 consensus_meta.cc:287] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.580s user 0.009s sys 0.002s
I20260808 01:11:19.079722 1517 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:19.065614 1442 raft_consensus.cc:1226] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:19.098212 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:19.098959 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:19.099591 1440 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 9.
I20260808 01:11:19.099625 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:19.102293 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:19.102924 2203 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 8.
I20260808 01:11:19.108793 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:19.109944 1410 leader_election.cc:398] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 6. Current term is 7.
I20260808 01:11:19.110394 1410 leader_election.cc:401] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:11:19.121433 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:19.121696 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260808 01:11:19.122107 1445 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:11:19.124279 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:11:19.126034 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:11:18.962219 4805 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:19.127069 1525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2073) took 1231 ms. Trace:
I20260808 01:11:19.126917 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.133332 1517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58250 (request call id 1081) took 1191 ms. Trace:
I20260808 01:11:19.130968 1525 rpcz_store.cc:276] 0808 01:11:17.895411 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:17.936083 (+ 40672us) service_pool.cc:227] Handling call
0808 01:11:17.945151 (+ 9068us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:18.120020 (+174869us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:19.126843 (+1006823us) spinlock_profiling.cc:243] Waited 5.2 ms on lock 0x7b54001a04d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:19.127013 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11968256}
I20260808 01:11:19.133845 1517 rpcz_store.cc:276] 0808 01:11:17.951151 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:19.124063 (+1172912us) service_pool.cc:227] Handling call
0808 01:11:19.133255 (+ 9192us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:19.140061 1328 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet a83a963773094d6db6495b7009fd8690 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6b09dbefd24b4d52b385c4dff5697dc5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:19.103222 4743 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:19.157404 1521 raft_consensus.cc:1261] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260808 01:11:19.159473 1444 raft_consensus.cc:2454] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
W20260808 01:11:19.160830 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 198433us
I20260808 01:11:19.161082 1410 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:19.161550 2171 leader_election.cc:334] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 1.739s (SENT)
I20260808 01:11:19.120664 4791 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:19.166244 2171 leader_election.cc:334] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 1.540s (SENT)
I20260808 01:11:19.079182 4401 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:19.166801 2171 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:19.169394 1445 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 10.
I20260808 01:11:19.102892 4713 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:19.180349 4593 raft_consensus.cc:2726] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 01:11:19.181842 4816 log.cc:924] Time spent T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
I20260808 01:11:19.181993 4713 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:19.173601 4812 raft_consensus.cc:2726] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 6. Current term is 7.
I20260808 01:11:19.044121 4654 consensus_queue.cc:237] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:19.174512 2171 leader_election.cc:334] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 1.782s (SENT)
I20260808 01:11:19.189654 2171 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:19.195781 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.202095 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.205381 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.207906 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.210739 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:19.211392 2209 raft_consensus.cc:1261] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:11:19.211585 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.212940 1517 raft_consensus.cc:2354] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for earlier term 9. Current term is 10.
I20260808 01:11:19.214382 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:19.215179 2207 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Advancing to term 10
I20260808 01:11:19.218587 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
W20260808 01:11:19.219647 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 140391us
I20260808 01:11:19.220697 1408 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:19.222500 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.197440 4812 raft_consensus.cc:2726] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:11:19.197631 4461 raft_consensus.cc:2726] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:19.163874 4743 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:19.230194 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.267249 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.267802 2209 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 7
I20260808 01:11:19.268314 1521 raft_consensus.cc:2354] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for earlier term 6. Current term is 7.
I20260808 01:11:19.201509 4740 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:19.269450 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:19.270584 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:19.271556 1443 raft_consensus.cc:1226] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:19.223104 1525 raft_consensus.cc:2454] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:11:19.274430 2207 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 10.
I20260808 01:11:19.286687 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.290935 2209 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 7.
I20260808 01:11:19.225783 2171 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:11:19.285842 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 60369.1us
I20260808 01:11:19.296830 1521 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 5: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:19.300227 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:19.302026 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.302649 1517 raft_consensus.cc:2454] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:19.304637 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.278956 4740 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:19.306417 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 80749us
I20260808 01:11:19.306882 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:19.309767 1521 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 5: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:19.312155 1516 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:19.289021 4823 raft_consensus.cc:2741] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260808 01:11:19.321718 1521 raft_consensus.cc:3037] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:11:19.329510 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 188651us
I20260808 01:11:19.270149 4733 consensus_queue.cc:1046] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.039s
I20260808 01:11:19.338188 1525 raft_consensus.cc:1226] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260808 01:11:19.339393 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:19.276404 4821 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.044s
I20260808 01:11:19.303823 4763 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:19.309293 4461 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:19.316993 4593 raft_consensus.cc:2781] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 01:11:19.307469 1393 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 44326.1us
I20260808 01:11:19.352885 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.367000 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.365161 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:19.367000 1521 raft_consensus.cc:1261] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:11:19.369330 1443 raft_consensus.cc:2396] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:11:19.320724 4810 consensus_queue.cc:1046] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.231s
I20260808 01:11:19.301178 4811 consensus_queue.cc:1035] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.265s
I20260808 01:11:19.373104 1446 raft_consensus.cc:1261] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:11:19.362519 4593 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:19.341954 4636 consensus_queue.cc:1035] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.031s
I20260808 01:11:19.413887 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:19.414601 4743 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:19.415423 4740 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:19.416306 4840 raft_consensus.cc:3032] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:11:19.421423 2208 raft_consensus.cc:1261] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260808 01:11:19.348477 4805 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:19.429973 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:19.310913 4833 consensus_queue.cc:1046] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.114s
W20260808 01:11:19.418203 4818 log.cc:924] Time spent T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.309s user 0.001s sys 0.000s
I20260808 01:11:19.419202 4401 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:19.422969 4593 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:19.367771 4461 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:19.373111 4812 consensus_queue.cc:1035] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.073s
I20260808 01:11:19.374850 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.479296 4642 raft_consensus.cc:3037] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:11:19.446619 4840 raft_consensus.cc:729] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:19.503515 4713 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:19.507769 4688 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:19.508177 4713 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:19.510303 4713 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:19.493136 4837 raft_consensus.cc:3032] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:19.511885 4837 raft_consensus.cc:729] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
W20260808 01:11:19.513516 4834 log.cc:924] Time spent T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.206s user 0.000s sys 0.001s
I20260808 01:11:19.515133 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.520560 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.519618 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:19.531927 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 165312us
W20260808 01:11:19.533849 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 113186us
W20260808 01:11:19.545809 4591 consensus_meta.cc:287] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.678s user 0.014s sys 0.000s
I20260808 01:11:19.548672 1524 raft_consensus.cc:2454] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
W20260808 01:11:19.553654 4824 log.cc:924] Time spent T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.161s user 0.001s sys 0.000s
W20260808 01:11:19.554600 4830 log.cc:924] Time spent T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.253s user 0.000s sys 0.001s
W20260808 01:11:19.564617 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 197575us
I20260808 01:11:19.512758 4837 consensus_queue.cc:260] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:19.507261 4840 consensus_queue.cc:260] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 6.9, Last appended by leader: 9, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:19.578370 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.552600 4728 consensus_queue.cc:1035] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260808 01:11:19.589322 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.590932 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:19.592373 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:19.572636 4480 consensus_meta.cc:287] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.703s user 0.012s sys 0.000s
I20260808 01:11:19.596092 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:19.599934 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.601846 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.604626 4293 consensus_queue.cc:1035] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 01:11:19.609177 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:19.611191 4649 consensus_meta.cc:287] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.686s user 0.008s sys 0.003s
I20260808 01:11:19.595023 4461 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:19.615891 4480 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:19.586380 4837 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:19.620270 4591 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:19.623704 1447 raft_consensus.cc:1226] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:19.598289 4755 consensus_queue.cc:1046] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20260808 01:11:19.625301 1516 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:19.646654 1525 raft_consensus.cc:1261] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:11:19.646800 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.647620 1516 raft_consensus.cc:2379] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:19.650221 1445 raft_consensus.cc:1261] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260808 01:11:19.628885 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:19.651865 1446 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:19.630086 1521 raft_consensus.cc:2396] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for term 8 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260808 01:11:19.653501 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:19.654215 1521 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:19.634500 1490 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:19.638887 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.657027 1517 raft_consensus.cc:2454] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 7.
I20260808 01:11:19.653204 2172 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:19.668561 1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2330) took 1149 ms. Trace:
I20260808 01:11:19.668992 1598 rpcz_store.cc:276] 0808 01:11:18.496442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:18.549538 (+ 53096us) service_pool.cc:227] Handling call
0808 01:11:19.645955 (+1096417us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b5400304654. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:19.646092 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38865280}
I20260808 01:11:19.668565 4591 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:19.673157 4591 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:19.673835 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.674988 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.675530 1517 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:19.631213 4835 consensus_queue.cc:1035] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260808 01:11:19.670574 4713 raft_consensus.cc:2726] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:19.671672 4847 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:19.685640 4847 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:19.689484 4847 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:19.691967 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:19.692677 1444 raft_consensus.cc:2362] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6. Re-sending same reply.
I20260808 01:11:19.694444 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:19.694792 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:19.695020 4823 raft_consensus.cc:3032] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:19.695564 4823 raft_consensus.cc:729] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:19.696084 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:19.696702 4823 consensus_queue.cc:260] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:19.698130 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:19.698403 4823 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:19.698271 4593 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:19.700392 4593 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:19.700291 1519 raft_consensus.cc:2454] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:19.701103 4461 raft_consensus.cc:2781] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:19.701540 4461 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:19.701897 4461 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:11:19.704264 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 289135us
I20260808 01:11:19.705751 1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2329) took 1209 ms. Trace:
I20260808 01:11:19.711709 2171 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:19.713742 1490 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:19.715054 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:19.715971 2207 raft_consensus.cc:1261] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEARNER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:11:19.716436 1490 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:19.717533 1604 raft_consensus.cc:2354] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5.
I20260808 01:11:19.706218 1596 rpcz_store.cc:276] 0808 01:11:18.495981 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:18.530691 (+ 34710us) service_pool.cc:227] Handling call
0808 01:11:18.533779 (+ 3088us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:18.603462 (+ 69683us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:18.603507 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:18.604139 (+ 632us) log.cc:842] Serialized 26 byte log entry
0808 01:11:18.840406 (+236267us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:18.962890 (+122484us) log.cc:842] Serialized 14 byte log entry
0808 01:11:18.963502 (+ 612us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:19.646485 (+682983us) spinlock_profiling.cc:243] Waited 69.4 ms on lock 0x7b54003908d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:19.646540 (+ 55us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:19.646596 (+ 56us) raft_consensus.cc:1671] finished
0808 01:11:19.705503 (+ 58907us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:19.705681 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":330991,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":63,"raft.queue_time_us":39430,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":22462,"spinlock_wait_cycles":159570176,"thread_start_us":288651,"threads_started":2,"wal-append.queue_time_us":328895}
I20260808 01:11:19.720814 1490 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:19.729247 1490 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:19.730355 1490 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:19.730441 4629 consensus_queue.cc:1046] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:19.731387 4642 raft_consensus.cc:2781] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:19.733354 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.722291 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.725229 2207 raft_consensus.cc:1261] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [term 8 LEARNER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260808 01:11:19.737038 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:19.737847 1516 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 5: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:19.738370 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.737082 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:19.747205 1522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2080) took 1834 ms. Trace:
I20260808 01:11:19.739094 2207 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:19.725229 4678 raft_consensus.cc:2726] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:11:19.747596 1522 rpcz_store.cc:276] 0808 01:11:17.913005 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:17.952665 (+ 39660us) service_pool.cc:227] Handling call
0808 01:11:17.978117 (+ 25452us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:18.931213 (+953096us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:19.746960 (+815747us) spinlock_profiling.cc:243] Waited 949 ms on lock 0x7b540023c3d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:19.747143 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2181609728}
I20260808 01:11:19.745748 1521 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:19.642069 4649 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:19.727267 4713 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:11:19.755512 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:19.731823 4642 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:19.764303 2203 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:19.740806 4591 raft_consensus.cc:2781] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:19.767948 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.769027 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:19.773243 1445 raft_consensus.cc:1261] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:11:19.774119 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:19.731707 4629 consensus_queue.cc:1035] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260808 01:11:19.788650 1410 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:19.724677 4844 consensus_queue.cc:1035] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.035s
I20260808 01:11:19.790407 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:19.785710 4568 consensus_queue.cc:1035] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260808 01:11:19.792814 1567 connection.cc:232] RPC call timeout handler was delayed by -1.2004390119982418s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17320000000000002s for a response
I20260808 01:11:19.770061 4642 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:19.795250 1410 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:19.801029 1520 raft_consensus.cc:2362] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4. Re-sending same reply.
I20260808 01:11:19.864424 1520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58280 (request call id 2107) took 1334 ms. Trace:
I20260808 01:11:19.864858 1520 rpcz_store.cc:276] 0808 01:11:18.548406 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:18.831904 (+283498us) service_pool.cc:227] Handling call
0808 01:11:19.784526 (+952622us) spinlock_profiling.cc:243] Waited 2.03 ms on lock 0x7b5800035d98. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adf851 00007fe349adf702 00007fe3529cfb09 00007fe35190b730 00007fe3519d179b 00007fe3519d1821 00007fe3519811b0 00007fe351980a94 00007fe34b068c04 00007fe34b068b92
0808 01:11:19.864327 (+ 79801us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4678912}
I20260808 01:11:19.765381 4517 raft_consensus.cc:3032] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:11:19.861768 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:19.873581 1521 raft_consensus.cc:1226] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 8. Current term is 9. Ops: [8.6-8.6]
W20260808 01:11:19.892576 1567 connection.cc:232] RPC call timeout handler was delayed by -1.0384390119870659s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1912s for a response
I20260808 01:11:19.778057 4688 raft_consensus.cc:2726] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 01:11:19.895336 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 100187us
I20260808 01:11:19.897245 1517 raft_consensus.cc:1261] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260808 01:11:19.903493 1410 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:19.905097 1410 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:19.907398 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:19.908308 1602 raft_consensus.cc:2454] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:19.778380 4293 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:19.858166 4835 raft_consensus.cc:2781] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:19.917204 3772 raft_consensus.cc:2726] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:11:19.919387 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:19.921754 3772 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:19.922392 3772 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:19.929195 1597 raft_consensus.cc:1261] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260808 01:11:19.935508 3772 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:19.896565 4553 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:19.939226 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:19.940421 1598 raft_consensus.cc:1226] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 8. Current term is 9. Ops: []
I20260808 01:11:19.940872 1600 raft_consensus.cc:1226] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:19.941170 1599 raft_consensus.cc:1226] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 8. Current term is 9. Ops: []
I20260808 01:11:19.893231 4517 raft_consensus.cc:729] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:19.913193 4847 raft_consensus.cc:2726] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:19.909907 4805 raft_consensus.cc:2726] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 01:11:19.958722 4850 log.cc:924] Time spent T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23: Append to log took a long time: real 0.170s user 0.001s sys 0.001s
W20260808 01:11:19.960449 4849 log.cc:924] Time spent T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.173s user 0.001s sys 0.000s
I20260808 01:11:19.961647 2201 raft_consensus.cc:1261] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEARNER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260808 01:11:19.959429 4517 consensus_queue.cc:260] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:19.933362 4835 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:19.968830 4517 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:19.990964 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:20.030812 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:20.032490 1599 raft_consensus.cc:3037] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:11:20.032678 2171 leader_election.cc:334] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 1.681s (SENT)
I20260808 01:11:20.033324 2171 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:20.033943 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 62756.2us
W20260808 01:11:20.034727 2171 leader_election.cc:334] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 1.823s (SENT)
I20260808 01:11:20.035305 2171 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:19.992026 1596 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:19.938877 4571 consensus_queue.cc:1035] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
I20260808 01:11:19.934772 4293 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:20.088793 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 358460us
I20260808 01:11:20.185637 1490 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:20.066315 4844 raft_consensus.cc:2726] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:19.975716 4835 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:11:20.145546 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 93034.8us
I20260808 01:11:19.996214 4545 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:11:20.154577 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 119874us
I20260808 01:11:19.920255 4825 consensus_queue.cc:1046] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.111s
I20260808 01:11:20.066911 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:20.054761 4846 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:19.968534 4480 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:20.155640 4785 raft_consensus.cc:2726] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:20.190891 4664 raft_consensus.cc:2726] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:20.196964 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:11:20.209671 2171 leader_election.cc:334] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 1.823s (SENT)
W20260808 01:11:20.233799 2171 leader_election.cc:334] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 1.843s (SENT)
I20260808 01:11:20.234254 2171 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:20.233909 2206 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.007s sys 0.007s
W20260808 01:11:20.235188 2206 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.506s user 0.007s sys 0.007s
W20260808 01:11:20.236001 2171 leader_election.cc:334] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 1.856s (SENT)
W20260808 01:11:20.231243 1601 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2621) took 1483 ms (client timeout 1713 ms). Trace:
I20260808 01:11:20.238664 2171 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:20.241082 1567 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:20.099195 4652 consensus_queue.cc:1035] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.116s
W20260808 01:11:20.241918 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 279291us
W20260808 01:11:20.237627 1601 rpcz_store.cc:269] 0808 01:11:18.747488 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:18.747629 (+ 141us) service_pool.cc:227] Handling call
0808 01:11:20.230987 (+1483358us) spinlock_profiling.cc:243] Waited 344 ms on lock 0x7b540028f5d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:20.231170 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":790668672}
I20260808 01:11:20.247108 1487 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:20.255548 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:11:20.257820 1487 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:20.259214 1487 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:20.273380 2201 raft_consensus.cc:1261] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:11:20.231560 4649 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:20.196242 4713 raft_consensus.cc:686] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
W20260808 01:11:20.297034 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 392011us
I20260808 01:11:20.247371 4733 raft_consensus.cc:2726] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:20.256335 4825 raft_consensus.cc:2781] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:20.298873 4597 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:20.252847 4844 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:20.230381 4269 consensus_queue.cc:1035] T 0108cd20b4bb4957b8a59eebd68479e4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260808 01:11:20.247040 4654 consensus_queue.cc:1035] T 1355c134ca5f45b381fb2b12b1865adb P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260808 01:11:20.295135 4689 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:20.218737 4846 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:20.313900 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 45629.5us
I20260808 01:11:20.314462 1410 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:20.249300 4857 consensus_queue.cc:1035] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260808 01:11:20.341773 4857 consensus_queue.cc:1035] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:20.319686 4865 raft_consensus.cc:2726] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:11:20.308825 4853 log.cc:924] Time spent T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.396s user 0.001s sys 0.000s
I20260808 01:11:20.254427 4785 raft_consensus.cc:2726] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:20.250972 4591 raft_consensus.cc:686] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
W20260808 01:11:20.327611 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 292822us
I20260808 01:11:20.334867 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:20.310315 4861 raft_consensus.cc:2726] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:20.350317 1525 raft_consensus.cc:1261] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:11:20.351594 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:20.356333 1521 raft_consensus.cc:1261] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:11:20.355511 1410 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:20.322111 4825 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:20.354266 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:20.321822 4870 raft_consensus.cc:2781] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:20.361264 3772 raft_consensus.cc:2726] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:20.362582 2201 raft_consensus.cc:1261] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:11:20.363372 1567 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:20.336031 4295 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:20.363271 1565 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:20.368535 1567 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:20.369390 4629 raft_consensus.cc:2726] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:20.370607 1567 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:20.371368 4629 raft_consensus.cc:2726] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:20.350792 4757 raft_consensus.cc:2726] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 01:11:20.392576 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 116543us
W20260808 01:11:20.393399 2171 leader_election.cc:334] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 1.738s (SENT)
I20260808 01:11:20.393942 2171 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:20.397915 3772 consensus_queue.cc:1035] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:11:20.400098 3772 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:20.400671 3772 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:20.391434 4591 consensus_queue.cc:237] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:20.403231 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:20.391927 4295 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:20.364708 4825 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:20.367770 4568 raft_consensus.cc:2781] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:20.336601 4597 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:20.420790 3772 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:20.426863 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 66819us
I20260808 01:11:20.433104 4295 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:20.436088 4649 raft_consensus.cc:2781] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:20.436569 4649 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:11:20.435173 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 72041.3us
I20260808 01:11:20.373344 4293 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:20.361186 4480 consensus_queue.cc:1035] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.057s
I20260808 01:11:20.453786 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:20.404994 4870 raft_consensus.cc:686] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:20.468501 1517 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 8.
I20260808 01:11:20.476356 1565 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:20.477545 1565 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:20.478819 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:20.436918 4649 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:20.482276 1565 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:20.479749 1525 raft_consensus.cc:2454] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:20.494742 4871 raft_consensus.cc:2726] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260808 01:11:20.495750 4820 log.cc:924] Time spent T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 1.361s user 0.005s sys 0.012s
W20260808 01:11:20.497773 4855 log.cc:924] Time spent T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260808 01:11:20.499908 4867 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:20.500545 4823 consensus_meta.cc:287] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.802s user 0.011s sys 0.004s
I20260808 01:11:20.500689 4867 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:20.503417 4461 consensus_meta.cc:287] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.801s user 0.012s sys 0.000s
I20260808 01:11:20.421645 4568 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:20.477629 4654 raft_consensus.cc:2781] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:20.373375 1567 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:20.505540 1565 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:20.535097 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 161804us
W20260808 01:11:20.542855 1524 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58266 (request call id 2211) took 2483 ms (client timeout 2648 ms). Trace:
W20260808 01:11:20.543339 1524 rpcz_store.cc:269] 0808 01:11:18.059404 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:19.126675 (+1067271us) service_pool.cc:227] Handling call
0808 01:11:20.542688 (+1416013us) spinlock_profiling.cc:243] Waited 416 ms on lock 0x7b540012f054. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:20.542794 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":957402880}
I20260808 01:11:20.544078 4401 raft_consensus.cc:3032] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:20.544579 4401 raft_consensus.cc:729] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
W20260808 01:11:20.545051 2205 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 721) took 3114 ms (client timeout 1679 ms). Trace:
I20260808 01:11:20.545614 4401 consensus_queue.cc:260] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:20.547062 4401 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:20.549379 4269 raft_consensus.cc:2726] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:11:20.537871 1567 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:20.454422 4732 raft_consensus.cc:2726] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260808 01:11:20.545434 2205 rpcz_store.cc:269] 0808 01:11:17.431060 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:17.431186 (+ 126us) service_pool.cc:227] Handling call
0808 01:11:20.544780 (+3113594us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b54005226d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:20.544938 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3819879808}
I20260808 01:11:20.556684 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:20.562844 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 57383.1us
I20260808 01:11:20.404714 4713 consensus_queue.cc:237] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:20.526821 4642 consensus_meta.cc:287] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.624s user 0.012s sys 0.000s
I20260808 01:11:20.574556 1567 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:20.518129 4571 raft_consensus.cc:2726] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:20.554185 4568 raft_consensus.cc:3037] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:20.503756 4867 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:20.582826 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:20.584350 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:20.554499 4654 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:20.586385 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:20.586736 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:20.591212 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 197834us
W20260808 01:11:20.607334 2171 leader_election.cc:334] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 1.788s (SENT)
I20260808 01:11:20.607905 2171 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:20.610016 2171 leader_election.cc:334] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 1.947s (SENT)
W20260808 01:11:20.611049 2171 leader_election.cc:334] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 1.976s (SENT)
I20260808 01:11:20.611510 2171 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:20.571218 4629 raft_consensus.cc:2726] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:20.581001 4731 consensus_queue.cc:1035] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
W20260808 01:11:20.614799 2171 leader_election.cc:334] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 2.070s (SENT)
I20260808 01:11:20.615653 2200 raft_consensus.cc:2454] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:11:20.618815 1523 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:20.721755 2171 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:20.514218 4461 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:20.742643 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 161162us
I20260808 01:11:20.587664 4652 raft_consensus.cc:2781] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader election won for term 9
W20260808 01:11:20.834582 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 219778us
I20260808 01:11:20.587962 4269 raft_consensus.cc:2781] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election won for term 5
W20260808 01:11:20.837430 2171 leader_election.cc:334] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 2.079s (SENT)
I20260808 01:11:20.997661 2171 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:20.872447 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 451709us
W20260808 01:11:20.917323 4873 log.cc:924] Time spent T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.360s user 0.001s sys 0.000s
I20260808 01:11:20.933535 4571 consensus_queue.cc:1035] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:11:20.581348 4642 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:20.613497 4654 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:20.924686 4597 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:20.839175 4629 consensus_queue.cc:1035] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.094s
I20260808 01:11:20.615260 4872 raft_consensus.cc:2726] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:21.000476 4295 consensus_queue.cc:1035] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:21.009537 4866 consensus_queue.cc:1046] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:11:21.107178 4866 raft_consensus.cc:3032] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:11:21.107685 4866 raft_consensus.cc:729] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:20.947269 4652 raft_consensus.cc:686] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 9 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:21.108803 4866 consensus_queue.cc:260] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:21.110555 4866 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:21.110065 4652 consensus_queue.cc:237] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:20.739202 4813 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:21.112630 4835 consensus_meta.cc:287] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.901s user 0.009s sys 0.003s
I20260808 01:11:20.742164 4857 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:21.079814 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 242428us
I20260808 01:11:21.113205 4835 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.113526 4857 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:20.751580 4881 raft_consensus.cc:2726] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:21.090911 4636 raft_consensus.cc:2726] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:21.094255 4461 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:21.096493 4867 consensus_queue.cc:1035] T b316a05b1f7448158ae76530ee0ee127 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:11:20.943096 4740 raft_consensus.cc:2726] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:20.945426 4846 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:20.740262 4870 consensus_queue.cc:237] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.111979 4813 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.082294 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:21.157007 4840 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:11:21.126446 2171 leader_election.cc:334] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 2.178s (SENT)
I20260808 01:11:21.140532 1522 raft_consensus.cc:2379] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 5: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:21.163931 2171 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:21.031785 4269 raft_consensus.cc:686] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:21.145744 1446 raft_consensus.cc:1226] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 7. Current term is 8. Ops: []
I20260808 01:11:21.175392 1524 raft_consensus.cc:2454] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:21.177791 1447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2271) took 1207 ms. Trace:
W20260808 01:11:21.177932 4517 consensus_meta.cc:287] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.209s user 0.012s sys 0.001s
I20260808 01:11:21.179801 4269 consensus_queue.cc:237] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:21.178363 1447 rpcz_store.cc:276] 0808 01:11:19.970219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:19.970484 (+ 265us) service_pool.cc:227] Handling call
0808 01:11:20.085856 (+115372us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:20.086254 (+ 398us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:20.086285 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:20.104895 (+ 18610us) log.cc:842] Serialized 26 byte log entry
0808 01:11:20.354039 (+249144us) raft_consensus.cc:1631] Marking committed up to 3
0808 01:11:20.354075 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:20.354141 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:21.177552 (+823411us) raft_consensus.cc:1671] finished
0808 01:11:21.177587 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:21.177727 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":842,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":60055,"thread_start_us":248779,"threads_started":1,"wal-append.queue_time_us":366070}
W20260808 01:11:21.191466 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 65043.5us
W20260808 01:11:21.192480 2171 leader_election.cc:334] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 2.231s (SENT)
I20260808 01:11:21.192921 2171 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:21.194214 2171 leader_election.cc:334] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 2.246s (SENT)
W20260808 01:11:21.196098 1517 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.004s sys 0.052s
W20260808 01:11:21.196645 1517 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.630s user 0.004s sys 0.052s
W20260808 01:11:21.197751 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 454135us
W20260808 01:11:21.184098 4553 consensus_meta.cc:287] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 1.219s user 0.012s sys 0.002s
I20260808 01:11:21.194687 2171 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:21.213589 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:11:21.220626 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 220351us
I20260808 01:11:21.222671 1410 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:21.227010 1410 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:21.196069 4881 raft_consensus.cc:2726] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:21.228178 1410 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:21.197180 4846 raft_consensus.cc:2726] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260808 01:11:21.231632 2171 leader_election.cc:334] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 2.255s (SENT)
I20260808 01:11:21.232499 2171 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:21.237372 1518 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
W20260808 01:11:21.217684 1520 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.781s user 0.009s sys 0.019s
I20260808 01:11:21.228708 4887 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:21.241737 2171 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:21.241896 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 219369us
W20260808 01:11:21.242749 1520 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.806s user 0.009s sys 0.019s
I20260808 01:11:21.239722 1567 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:21.228991 4295 raft_consensus.cc:2781] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:11:21.225701 4635 raft_consensus.cc:2726] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:21.261276 3772 raft_consensus.cc:2781] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:21.264638 3772 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:21.265077 3772 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:11:21.262568 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 53995.9us
I20260808 01:11:21.220897 4871 consensus_queue.cc:1035] T a3ce33a519294ae1bce66c8aca8cf601 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260808 01:11:21.264181 2171 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:21.271322 2171 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:21.280031 1524 raft_consensus.cc:1261] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260808 01:11:21.245633 4881 raft_consensus.cc:2726] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:21.291501 3772 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:21.261886 4825 consensus_meta.cc:287] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.837s user 0.013s sys 0.000s
I20260808 01:11:21.260281 4846 raft_consensus.cc:2726] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260808 01:11:21.269213 4844 consensus_meta.cc:287] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.933s user 0.012s sys 0.002s
W20260808 01:11:21.293640 1567 leader_election.cc:334] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 2.031s (SENT)
I20260808 01:11:21.303541 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:21.351331 1567 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:21.352208 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 58629.5us
W20260808 01:11:21.352802 1567 leader_election.cc:334] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 2.077s (SENT)
I20260808 01:11:21.353170 1567 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:21.325721 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 54467.8us
W20260808 01:11:21.340353 1328 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet a83a963773094d6db6495b7009fd8690 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6b09dbefd24b4d52b385c4dff5697dc5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:21.289239 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:21.263911 4295 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:11:21.299392 1489 connection.cc:232] RPC call timeout handler was delayed by -1.5411585030087736s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1713s for a response
W20260808 01:11:21.359345 1489 connection.cc:232] RPC call timeout handler was delayed by -1.5285585030214861s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17270000000000002s for a response
I20260808 01:11:21.332433 1445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2267) took 1552 ms. Trace:
W20260808 01:11:21.360119 1489 connection.cc:232] RPC call timeout handler was delayed by -1.5285585030214861s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17270000000000002s for a response
W20260808 01:11:21.360469 1489 connection.cc:232] RPC call timeout handler was delayed by -1.3503585029975511s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1925s for a response
I20260808 01:11:21.277746 4517 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:11:21.360976 1489 connection.cc:232] RPC call timeout handler was delayed by -1.2063585029973183s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20850000000000002s for a response
I20260808 01:11:21.360271 1445 rpcz_store.cc:276] 0808 01:11:19.780440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:19.788024 (+ 7584us) service_pool.cc:227] Handling call
0808 01:11:19.788346 (+ 322us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:20.213057 (+424711us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:20.213100 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:20.213170 (+ 70us) raft_consensus.cc:1631] Marking committed up to 3
0808 01:11:20.213215 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:21.332106 (+1118891us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b54000e74d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:21.332167 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:21.332349 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":976205056}
I20260808 01:11:21.328470 4757 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:21.361531 1489 connection.cc:232] RPC call timeout handler was delayed by -1.1523585030226968s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21450000000000002s for a response
I20260808 01:11:21.354689 1524 raft_consensus.cc:2454] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:21.266798 4553 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:21.277933 4810 raft_consensus.cc:2741] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260808 01:11:21.363742 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:21.358839 4295 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:21.277303 4461 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:21.282392 4843 raft_consensus.cc:2726] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:11:21.363211 1489 connection.cc:232] RPC call timeout handler was delayed by -1.0020585030142684s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2312s for a response
I20260808 01:11:21.376168 1489 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:21.280583 4720 raft_consensus.cc:2726] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:11:21.328372 4831 log.cc:924] Time spent T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 1.854s user 0.003s sys 0.008s
I20260808 01:11:21.378587 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:21.277555 4688 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:21.318260 4649 consensus_meta.cc:287] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.797s user 0.012s sys 0.000s
I20260808 01:11:21.329180 4784 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:21.374439 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:21.376387 3772 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:21.267014 4740 consensus_queue.cc:1035] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.095s
I20260808 01:11:21.377561 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:21.369786 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:21.299633 4597 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:21.288555 4690 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:21.275411 4636 consensus_queue.cc:1035] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.092s
I20260808 01:11:21.353569 4635 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:21.379959 1525 raft_consensus.cc:2454] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 3.
I20260808 01:11:21.366079 4891 raft_consensus.cc:2726] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:11:21.261813 4887 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:21.393105 3772 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:21.345621 4825 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.366004 4871 raft_consensus.cc:2726] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:11:21.365093 4757 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:21.395130 2202 raft_consensus.cc:2379] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:21.290557 4629 consensus_queue.cc:1035] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.092s
I20260808 01:11:21.301730 4731 consensus_queue.cc:1035] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
I20260808 01:11:21.370839 4835 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:21.446831 3772 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:21.365779 4844 raft_consensus.cc:2726] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:21.389146 4642 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:21.435286 4713 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:11:21.476114 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 280635us
I20260808 01:11:21.392421 4842 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:21.375065 4553 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:21.476948 4886 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:21.452395 4896 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:21.382341 4461 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.511982 1438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2248) took 2621 ms. Trace:
I20260808 01:11:21.439317 4732 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:21.521068 3772 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:21.559552 3772 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:21.560176 3772 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:21.532811 1604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2619) took 2786 ms (client timeout 1925 ms). Trace:
I20260808 01:11:21.391390 4688 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:21.469000 4810 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:21.561496 1604 rpcz_store.cc:269] 0808 01:11:18.746832 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:18.746965 (+ 133us) service_pool.cc:227] Handling call
0808 01:11:21.532478 (+2785513us) spinlock_profiling.cc:243] Waited 888 ms on lock 0x7b54002dafd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:21.532661 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2042883456}
I20260808 01:11:21.572279 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:21.434973 4597 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:21.435317 4690 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.573210 2207 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:21.479862 4871 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:21.558540 1438 rpcz_store.cc:276] 0808 01:11:18.890553 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:18.890763 (+ 210us) service_pool.cc:227] Handling call
0808 01:11:18.909671 (+ 18908us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:18.919997 (+ 10326us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:18.920046 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:18.935776 (+ 15730us) log.cc:842] Serialized 26 byte log entry
0808 01:11:19.166501 (+230725us) raft_consensus.cc:1631] Marking committed up to 2
0808 01:11:19.166959 (+ 458us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:19.167031 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:20.566786 (+1399755us) spinlock_profiling.cc:243] Waited 5.38 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:21.484326 (+917540us) raft_consensus.cc:1671] finished
0808 01:11:21.484353 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:21.511880 (+ 27527us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":80992,"raft.queue_time_us":9990,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":9052,"spinlock_wait_cycles":12364544,"thread_start_us":125898,"threads_started":1,"wal-append.queue_time_us":213935}
I20260808 01:11:21.578812 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:21.484272 4835 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:21.398175 4649 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.586822 3772 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:21.560489 4842 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.594900 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:21.610594 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:21.614549 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:21.595940 1487 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:21.402407 4784 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.513159 4642 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:21.446450 4635 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.562484 4832 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:21.618398 3772 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:21.571342 4857 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:21.634994 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:21.620461 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:21.629249 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 253198us
I20260808 01:11:21.629555 1524 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:21.558990 4293 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:21.628280 3772 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.645506 1489 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:21.647241 1489 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:21.627497 4269 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:21.519788 4713 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:21.626528 4568 consensus_meta.cc:287] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.001s user 0.011s sys 0.003s
I20260808 01:11:21.613055 4901 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:21.662927 3772 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:21.705456 4899 log.cc:924] Time spent T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.146s user 0.000s sys 0.001s
I20260808 01:11:21.705675 4825 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:21.706630 4809 log.cc:924] Time spent T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 2.577s user 0.030s sys 0.033s
W20260808 01:11:21.707363 1518 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58280 (request call id 2104) took 3178 ms (client timeout 2077 ms). Trace:
W20260808 01:11:21.707877 1518 rpcz_store.cc:269] 0808 01:11:18.528728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:18.687809 (+159081us) service_pool.cc:227] Handling call
0808 01:11:21.707056 (+3019247us) spinlock_profiling.cc:243] Waited 2.53 s on lock 0x7b54001a1654. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:21.707208 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5825387264}
W20260808 01:11:21.710642 1516 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58266 (request call id 2223) took 2078 ms (client timeout 2081 ms). Trace:
I20260808 01:11:21.710924 4635 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:21.711130 1516 rpcz_store.cc:269] 0808 01:11:19.632663 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:19.738188 (+105525us) service_pool.cc:227] Handling call
0808 01:11:21.710400 (+1972212us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b540035c0d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:21.710568 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3812893056}
I20260808 01:11:21.714988 4901 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.578884 4896 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:21.580211 4732 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.662187 4269 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.613459 4871 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.727823 1522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58266 (request call id 2227) took 2079 ms. Trace:
I20260808 01:11:21.728312 1522 rpcz_store.cc:276] 0808 01:11:19.648171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:19.756934 (+108763us) service_pool.cc:227] Handling call
0808 01:11:21.727601 (+1970667us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b540005d554. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:21.727758 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3147585280}
I20260808 01:11:21.695035 4651 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:21.730257 4871 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:21.729995 4651 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.732568 4688 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:21.733377 4833 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:21.733544 4688 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:21.734470 4412 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:21.617421 4835 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:21.734095 4833 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.735105 4412 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:21.736887 1519 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58280 (request call id 2100) took 3254 ms (client timeout 2031 ms). Trace:
I20260808 01:11:21.737370 4903 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:21.738155 4412 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:21.738023 4903 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:21.737397 1519 rpcz_store.cc:269] 0808 01:11:18.485022 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:18.490749 (+ 5727us) service_pool.cc:227] Handling call
0808 01:11:21.736667 (+3245918us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540035d4d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:21.736798 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2463741312}
I20260808 01:11:21.734479 4688 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:21.742594 4827 log.cc:924] Time spent T 31f1af2e7c504c919e34f12663452f2a P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 2.569s user 0.009s sys 0.037s
I20260808 01:11:21.742666 4688 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:21.743691 4688 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:21.746809 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:21.747814 1516 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:21.751930 1524 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927"
dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
mode: GRACEFUL
new_leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
from {username='slave'} at 127.0.0.1:48844
I20260808 01:11:21.752640 1524 raft_consensus.cc:595] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Received request to transfer leadership to TS 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:21.753026 1524 raft_consensus.cc:603] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:11:21.645532 4857 raft_consensus.cc:2726] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260808 01:11:21.662566 4401 consensus_meta.cc:287] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.111s user 0.012s sys 0.006s
I20260808 01:11:21.582593 4810 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:21.772193 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:21.772403 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:21.773173 1442 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:21.768559 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:21.770197 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 369139us
I20260808 01:11:21.568112 4886 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:21.723551 4784 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:21.777971 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:21.788158 2202 raft_consensus.cc:2379] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 6: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:21.778137 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:21.784322 1524 raft_consensus.cc:1261] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:11:21.641263 4642 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:21.792148 2207 raft_consensus.cc:2379] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 7: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:21.792910 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:21.795728 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:21.744403 4688 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.797972 32593 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260808 01:11:21.654151 4293 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.799106 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:21.800168 32593 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260808 01:11:21.822602 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:11:21.802280 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:21.804291 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:21.806288 4886 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:21.807952 4907 raft_consensus.cc:2726] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:21.808691 4461 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:21.810400 4810 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:21.838424 1439 raft_consensus.cc:2379] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 7: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:21.766345 4640 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:21.926467 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 43656us
W20260808 01:11:21.927130 4654 consensus_meta.cc:287] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.792s user 0.003s sys 0.012s
I20260808 01:11:21.755492 4732 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:21.927614 4654 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.928192 4845 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:21.928311 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:21.929109 4845 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.930135 4654 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:21.931224 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:21.850246 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:21.932186 1444 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:21.937520 4845 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:21.849378 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:21.859187 4811 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:21.860802 1517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36560 (request call id 654) took 1381 ms. Trace:
I20260808 01:11:21.772734 4846 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:21.939420 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:21.939611 4811 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.940573 2205 raft_consensus.cc:2379] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 6: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:21.939570 1517 rpcz_store.cc:276] 0808 01:11:20.478954 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:20.479215 (+ 261us) service_pool.cc:227] Handling call
0808 01:11:20.523163 (+ 43948us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:20.523373 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 01:11:21.205103 (+681730us) log.cc:842] Serialized 14 byte log entry
0808 01:11:21.856488 (+651385us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:21.860064 (+ 3576us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:21.860224 (+ 160us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:21.860394 (+ 170us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:21.860675 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64780,"prepare.queue_time_us":872153,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":151274,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":13086,"thread_start_us":1289632,"threads_started":3,"wal-append.queue_time_us":635696}
I20260808 01:11:21.942230 4811 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:21.869096 4690 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:21.840649 4757 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:21.870065 4553 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:21.944156 2207 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:21.871173 4651 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:21.876726 4842 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:21.877856 4887 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:21.645675 4832 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:21.946578 1408 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:21.946933 4832 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:21.879931 4642 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:21.696429 4568 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:21.949011 4568 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:21.881433 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 234264us
I20260808 01:11:21.837570 1448 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:21.898622 4905 raft_consensus.cc:2781] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:21.899535 4293 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:21.899662 4903 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:21.900455 4597 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:21.900600 4866 consensus_meta.cc:287] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.789s user 0.018s sys 0.001s
I20260808 01:11:21.881603 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:21.914028 1442 raft_consensus.cc:2454] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:21.939960 4846 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:21.785439 4870 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.001s sys 0.027s
I20260808 01:11:21.903751 4907 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:21.951048 2172 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:21.969458 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:21.950881 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:21.952538 1490 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:21.819852 4269 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:21.962183 1567 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:21.965471 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:21.950389 1489 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:21.904245 4910 raft_consensus.cc:2726] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:21.950271 4688 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:21.959539 4905 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:21.979758 1597 raft_consensus.cc:1226] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:21.981637 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:21.957934 4887 raft_consensus.cc:2781] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260808 01:11:21.963042 4909 log.cc:924] Time spent T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.192s user 0.002s sys 0.000s
I20260808 01:11:21.987270 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:21.988478 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:21.989423 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:21.978044 1600 raft_consensus.cc:1226] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
W20260808 01:11:21.966199 4870 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.782s user 0.001s sys 0.028s
W20260808 01:11:21.967636 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 190299us
I20260808 01:11:21.995491 1489 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:22.041388 1603 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:22.041917 1599 raft_consensus.cc:2396] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260808 01:11:22.032686 1602 raft_consensus.cc:2379] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 7: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:22.046924 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:21.994534 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.048213 1598 raft_consensus.cc:2354] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 8. Current term is 9.
I20260808 01:11:21.993553 1444 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:22.050087 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.050797 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.051045 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.051894 1602 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:22.052788 1410 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:22.052939 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:21.985019 4295 consensus_meta.cc:287] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.601s user 0.012s sys 0.001s
I20260808 01:11:22.064965 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.064536 2172 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:22.064731 1410 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:22.072376 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.072762 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:21.994793 4269 consensus_queue.cc:1035] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.078s
I20260808 01:11:22.073544 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.073802 1602 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
W20260808 01:11:22.077750 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 82317.5us
I20260808 01:11:21.954980 4640 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:22.078794 1489 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:21.983145 4651 raft_consensus.cc:2726] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:21.988831 4842 raft_consensus.cc:2781] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:21.995424 4690 raft_consensus.cc:2726] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:22.088603 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.082252 1565 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:22.080907 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.089047 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.085590 1410 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:22.098897 1489 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:21.994905 4597 raft_consensus.cc:2726] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:21.964169 4568 consensus_queue.cc:1035] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.086s
W20260808 01:11:22.096169 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 500322us
I20260808 01:11:22.108397 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:22.109649 2207 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:22.110370 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:22.112447 4688 raft_consensus.cc:2726] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:11:22.113309 2171 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:22.113565 4929 raft_consensus.cc:2726] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:22.076252 4295 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:22.035091 4905 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:22.033857 4649 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:22.081036 4843 raft_consensus.cc:2781] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:22.080850 4832 raft_consensus.cc:2726] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:11:22.053215 4833 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:22.095772 4842 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:22.136622 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:22.137854 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:22.083056 4922 raft_consensus.cc:2781] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:22.146059 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.128988 1410 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:22.100463 4784 raft_consensus.cc:2726] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:21.984117 4907 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:22.108399 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:22.257320 2205 raft_consensus.cc:2454] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
W20260808 01:11:22.121141 4840 consensus_meta.cc:287] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.963s user 0.010s sys 0.006s
I20260808 01:11:22.181610 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:22.261811 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:22.262861 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:22.182722 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:22.263738 1597 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 6
W20260808 01:11:22.192229 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 50263.2us
I20260808 01:11:22.202665 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:22.203668 1603 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 8.
I20260808 01:11:22.100258 4757 consensus_queue.cc:1046] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.081s
I20260808 01:11:22.127633 4713 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:22.132191 4844 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:22.242568 1443 raft_consensus.cc:1226] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 8. Current term is 9. Ops: []
I20260808 01:11:22.149137 4636 raft_consensus.cc:2726] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260808 01:11:22.247406 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 132235us
I20260808 01:11:22.292348 4844 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:22.296550 4757 raft_consensus.cc:3032] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:11:22.297200 4757 raft_consensus.cc:729] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:22.268728 4870 raft_consensus.cc:2726] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:22.299889 4293 raft_consensus.cc:2781] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:22.300661 4293 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:22.301339 4293 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:22.120494 4651 raft_consensus.cc:2726] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:22.251966 2207 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:22.246932 4843 raft_consensus.cc:686] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:22.265730 1602 raft_consensus.cc:1226] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 7. Current term is 8. Ops: [6.7-7.8]
I20260808 01:11:22.271498 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:22.274696 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 158835us
I20260808 01:11:22.314706 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:22.278661 1598 raft_consensus.cc:1226] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:22.281466 4918 consensus_queue.cc:1035] T 44f40a08ace944a88ad5bf8ba22f2a40 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.186s
W20260808 01:11:22.284255 4923 log.cc:924] Time spent T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.250s user 0.000s sys 0.001s
I20260808 01:11:22.251998 2203 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:22.276886 2208 raft_consensus.cc:2396] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 7 because replica has last-logged OpId of term: 6 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20260808 01:11:22.315474 1524 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:22.321806 1410 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:22.321945 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:22.254508 4842 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:22.326917 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:22.253738 4934 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:22.328253 1567 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:22.254431 4922 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:22.328781 1524 raft_consensus.cc:2379] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 7: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:22.334368 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:22.338629 2205 raft_consensus.cc:2932] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 3 to 6, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:22.342980 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.343804 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.305490 4835 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:22.366529 1604 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:22.344386 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:22.369488 1565 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:22.346218 1442 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260808 01:11:22.348735 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:22.372287 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:22.373104 1524 raft_consensus.cc:3037] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:22.350227 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:22.351698 1490 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:22.352954 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.355805 1568 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:22.356052 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.376029 1596 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 7.
I20260808 01:11:22.367183 1598 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 7.
I20260808 01:11:22.382576 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.391175 1597 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:22.393712 4667 raft_consensus.cc:2726] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:11:22.394914 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:22.368362 4934 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:22.395938 4912 raft_consensus.cc:2781] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:22.396579 4912 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:22.397141 4912 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:22.398900 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.400153 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
W20260808 01:11:22.399542 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 43754.4us
I20260808 01:11:22.401654 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:22.402598 1604 raft_consensus.cc:2379] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 6: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:22.324739 4714 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:22.409215 2171 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:22.453351 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 44252.3us
W20260808 01:11:22.456036 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 208695us
I20260808 01:11:22.325313 4813 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:22.457549 1565 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:22.472344 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:22.474516 1439 raft_consensus.cc:2379] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 5: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
W20260808 01:11:22.514786 1522 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.033s sys 0.054s
W20260808 01:11:22.516012 1522 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s user 0.034s sys 0.054s
I20260808 01:11:22.302136 4871 raft_consensus.cc:2932] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 3 to 6, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:22.448590 4480 raft_consensus.cc:2726] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:11:22.372128 4810 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:22.527601 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 126301us
I20260808 01:11:22.371009 4866 raft_consensus.cc:2781] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:11:22.456730 4784 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:22.459646 4811 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:22.371526 4922 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:22.398511 4757 consensus_queue.cc:260] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:22.904026 1410 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.069s sys 0.138s
W20260808 01:11:22.988643 1603 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.038s sys 0.077s
I20260808 01:11:22.782083 1521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2235) took 2376 ms. Trace:
W20260808 01:11:23.087971 1603 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.688s user 0.041s sys 0.087s
W20260808 01:11:22.843295 2171 leader_election.cc:334] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 1.595s (SENT)
W20260808 01:11:23.089147 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 245903us
W20260808 01:11:23.087711 1410 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.730s user 0.095s sys 0.158s
W20260808 01:11:23.091545 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 733942us
I20260808 01:11:22.643993 1443 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2276) took 1068 ms. Trace:
W20260808 01:11:22.775039 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 317644us
I20260808 01:11:23.092695 1410 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:23.093679 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 312375us
I20260808 01:11:23.094870 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:23.049970 1522 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.000s sys 0.006s
I20260808 01:11:22.638476 1440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2278) took 1046 ms. Trace:
I20260808 01:11:23.096333 1446 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 788) took 2763 ms. Trace:
W20260808 01:11:23.096244 1522 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.563s user 0.001s sys 0.006s
W20260808 01:11:23.101125 1524 consensus_meta.cc:287] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.727s user 0.011s sys 0.000s
I20260808 01:11:22.400969 4688 raft_consensus.cc:2781] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:23.088227 1521 rpcz_store.cc:276] 0808 01:11:20.405527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:20.405653 (+ 126us) service_pool.cc:227] Handling call
0808 01:11:20.406123 (+ 470us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:20.487055 (+ 80932us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:20.487101 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:20.487158 (+ 57us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:20.487204 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:22.781749 (+2294545us) spinlock_profiling.cc:243] Waited 80.6 ms on lock 0x7b54000e9f54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:22.781810 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:22.781966 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":185484160}
W20260808 01:11:22.628057 2200 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1126) took 4137 ms (client timeout 2704 ms). Trace:
I20260808 01:11:23.105436 1487 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
W20260808 01:11:23.110838 2200 rpcz_store.cc:269] 0808 01:11:18.490232 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:18.490401 (+ 169us) service_pool.cc:227] Handling call
0808 01:11:22.627810 (+4137409us) spinlock_profiling.cc:243] Waited 2.12 s on lock 0x7b5400708fd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:22.627978 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4885588352}
I20260808 01:11:22.467481 4927 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:23.096563 1440 rpcz_store.cc:276] 0808 01:11:21.591644 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:21.591796 (+ 152us) service_pool.cc:227] Handling call
0808 01:11:21.617503 (+ 25707us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:21.804347 (+186844us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:21.804386 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:21.804424 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:21.804456 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:22.638145 (+833689us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54000e6ad4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:22.638202 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:22.638399 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":429230464}
I20260808 01:11:23.125159 1410 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:23.092741 1443 rpcz_store.cc:276] 0808 01:11:21.575224 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:21.575490 (+ 266us) service_pool.cc:227] Handling call
0808 01:11:21.575879 (+ 389us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:22.292338 (+716459us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:22.643653 (+351315us) spinlock_profiling.cc:243] Waited 666 ms on lock 0x7b54002c6fd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:22.643916 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1532643712}
W20260808 01:11:22.914034 4938 log.cc:924] Time spent T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.310s user 0.001s sys 0.000s
I20260808 01:11:23.123934 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:22.514938 4895 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:23.125191 1487 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:23.132802 1487 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:23.107952 1524 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:22.794414 4295 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 9 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:23.139433 2205 consensus_meta.cc:287] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.761s user 0.008s sys 0.004s
I20260808 01:11:23.097107 1446 rpcz_store.cc:276] 0808 01:11:20.332936 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:20.333086 (+ 150us) service_pool.cc:227] Handling call
0808 01:11:20.333471 (+ 385us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:21.178454 (+844983us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:22.747934 (+1569480us) spinlock_profiling.cc:243] Waited 808 ms on lock 0x7b540009e054. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:23.096252 (+348318us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1859242752}
W20260808 01:11:23.069325 4517 consensus_meta.cc:287] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.698s user 0.016s sys 0.000s
I20260808 01:11:22.462867 4714 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:22.669183 4920 log.cc:924] Time spent T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.263s user 0.001s sys 0.000s
I20260808 01:11:22.531999 4640 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:22.519404 4713 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:23.134912 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:23.136260 1524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58280 (request call id 2141) took 1147 ms. Trace:
I20260808 01:11:23.159550 1524 rpcz_store.cc:276] 0808 01:11:21.989389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:22.371978 (+382589us) service_pool.cc:227] Handling call
0808 01:11:23.136182 (+764204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:22.634601 4755 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:23.110010 4821 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:22.943159 4593 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:23.096736 4907 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:22.626456 4901 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:22.932871 4934 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:22.520582 4690 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:22.520356 4910 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:22.759600 4846 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:22.615510 4886 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:23.097723 4740 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:23.097846 4925 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:23.097810 4896 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:22.581305 4844 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:22.513880 4926 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:22.685374 4916 log.cc:924] Time spent T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.504s user 0.000s sys 0.001s
I20260808 01:11:22.779762 1447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 797) took 1011 ms. Trace:
I20260808 01:11:22.768769 4760 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:23.142868 1408 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.002s sys 0.010s
W20260808 01:11:23.255240 1408 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.730s user 0.004s sys 0.012s
I20260808 01:11:23.235550 4934 raft_consensus.cc:2781] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:23.256039 4934 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:23.254966 1447 rpcz_store.cc:276] 0808 01:11:21.768869 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:21.768986 (+ 117us) service_pool.cc:227] Handling call
0808 01:11:21.800013 (+ 31027us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:21.800234 (+ 221us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:21.800258 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:21.841261 (+ 41003us) log.cc:842] Serialized 26 byte log entry
0808 01:11:22.027035 (+185774us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:22.027093 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:22.027184 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:22.779481 (+752297us) raft_consensus.cc:1671] finished
0808 01:11:22.779511 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:22.779684 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":155626,"threads_started":1,"wal-append.queue_time_us":653637}
I20260808 01:11:23.256517 4934 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:23.154016 1410 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:23.112687 4866 raft_consensus.cc:686] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:22.775410 4651 raft_consensus.cc:2726] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:11:23.127707 4553 raft_consensus.cc:2781] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:22.528057 4843 consensus_queue.cc:237] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.131351 4688 raft_consensus.cc:686] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:23.129356 4835 raft_consensus.cc:2726] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:23.186486 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:23.161595 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:23.111898 4784 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.109953 4810 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:23.157407 4941 log.cc:924] Time spent T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.185s user 0.001s sys 0.001s
W20260808 01:11:23.158082 1600 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.023s user 0.000s sys 0.012s
I20260808 01:11:23.151358 4728 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:23.161330 1567 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:23.327962 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 166715us
I20260808 01:11:23.184566 4936 raft_consensus.cc:1050] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:11:23.159543 4295 raft_consensus.cc:2781] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:23.158926 4517 raft_consensus.cc:2726] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:23.159070 4269 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:11:23.216068 4914 log.cc:924] Time spent T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.641s user 0.002s sys 0.000s
I20260808 01:11:23.356909 1601 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:23.154246 4945 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:23.358939 4945 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.360152 2209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 977) took 3651 ms. Trace:
I20260808 01:11:23.132055 4927 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.364122 4927 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:23.219537 4733 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:23.365199 4810 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:23.365067 4733 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.367183 4784 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:23.156307 4629 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:23.236397 4930 log.cc:924] Time spent T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23: Append to log took a long time: real 0.869s user 0.001s sys 0.004s
I20260808 01:11:23.267974 4844 consensus_queue.cc:1046] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.087s
I20260808 01:11:23.367980 4629 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.368377 4714 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:23.274833 4942 log.cc:924] Time spent T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.280s user 0.001s sys 0.001s
I20260808 01:11:23.254556 4925 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.370504 4629 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:23.370966 1516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2124) took 2083 ms. Trace:
I20260808 01:11:23.185566 4755 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.354421 4728 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.251192 4846 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:23.374075 4755 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:23.374423 4728 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:23.374583 4846 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.371503 4925 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:23.375548 4269 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.377357 4846 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:23.263729 4926 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.377799 4269 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:23.378027 4293 consensus_meta.cc:287] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 1.076s user 0.014s sys 0.003s
I20260808 01:11:23.378373 4651 raft_consensus.cc:2781] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:23.378494 4293 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.254314 4740 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.381116 4740 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:23.251462 4886 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:23.236671 4690 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:23.283162 4905 consensus_meta.cc:287] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 1.046s user 0.015s sys 0.000s
I20260808 01:11:23.283245 1594 raft_consensus.cc:1226] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:23.283813 1525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2123) took 1997 ms. Trace:
I20260808 01:11:23.216696 4821 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.287638 4832 raft_consensus.cc:2726] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:23.266489 4760 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:23.239310 4907 raft_consensus.cc:2781] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:23.353323 4295 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
W20260808 01:11:23.290885 4888 log.cc:924] Time spent T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 2.685s user 0.013s sys 0.000s
W20260808 01:11:23.298064 4893 log.cc:924] Time spent T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 1.739s user 0.004s sys 0.000s
I20260808 01:11:23.183862 4713 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:23.302241 4921 log.cc:924] Time spent T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 1.061s user 0.000s sys 0.003s
I20260808 01:11:23.140522 4895 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:23.306262 4948 log.cc:924] Time spent T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.314s user 0.001s sys 0.000s
I20260808 01:11:23.215696 4593 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:23.323355 1600 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.188s user 0.000s sys 0.017s
I20260808 01:11:23.305192 4896 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:23.268234 4553 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:23.360626 2209 rpcz_store.cc:276] 0808 01:11:19.708729 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:19.708923 (+ 194us) service_pool.cc:227] Handling call
0808 01:11:19.709354 (+ 431us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:19.717662 (+ 8308us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:19.717705 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:19.726346 (+ 8641us) log.cc:842] Serialized 26 byte log entry
0808 01:11:21.457010 (+1730664us) spinlock_profiling.cc:243] Waited 1.61 ms on lock 0x7b1800393700. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349af9345 00007fe352930a39 00007fe349b168bb 00007fe349b14cbf 00007fe349b16e22 00007fe34c6c05bb 00007fe34c6c6063 00007fe34c6c6411 00007fe34ca135cf
0808 01:11:21.769943 (+312933us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:21.769992 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:21.770057 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:23.359746 (+1589689us) raft_consensus.cc:1671] finished
0808 01:11:23.359771 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:23.360057 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7824,"raft.queue_time_us":14857,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":8122,"spinlock_wait_cycles":3700992,"thread_start_us":312755,"threads_started":1,"wal-append.queue_time_us":2361213}
I20260808 01:11:23.367298 4733 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:23.261082 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:23.368516 4844 raft_consensus.cc:3032] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:11:23.278067 4866 consensus_queue.cc:237] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.379037 4651 raft_consensus.cc:686] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:23.237135 4910 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:23.361632 4945 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:23.284972 2208 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 727) took 3570 ms (client timeout 2231 ms). Trace:
W20260808 01:11:23.410259 4947 log.cc:924] Time spent T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.182s user 0.003s sys 0.000s
I20260808 01:11:23.404188 4905 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.396026 4926 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:23.413512 4907 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:23.465914 4907 raft_consensus.cc:3037] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:23.430729 4896 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:23.579737 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 306283us
W20260808 01:11:23.269073 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 115141us
I20260808 01:11:23.286406 1445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2274) took 1745 ms. Trace:
I20260808 01:11:23.581832 1446 raft_consensus.cc:1226] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:23.583629 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:23.279322 1488 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.703s user 0.032s sys 0.040s
I20260808 01:11:23.674906 1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 638) took 1604 ms. Trace:
W20260808 01:11:23.675634 1488 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.160s user 0.033s sys 0.040s
I20260808 01:11:23.675519 1602 rpcz_store.cc:276] 0808 01:11:22.070902 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:22.075388 (+ 4486us) service_pool.cc:227] Handling call
0808 01:11:22.265462 (+190074us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:22.312360 (+ 46898us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:23.674292 (+1361932us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54002ddcd0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:23.674642 (+ 350us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":436250496}
W20260808 01:11:23.677352 4931 log.cc:924] Time spent T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.733s user 0.006s sys 0.002s
W20260808 01:11:23.309927 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 177185us
I20260808 01:11:23.677747 1520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2236) took 3269 ms. Trace:
I20260808 01:11:23.371472 1516 rpcz_store.cc:276] 0808 01:11:21.287308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:21.758157 (+470849us) service_pool.cc:227] Handling call
0808 01:11:21.758566 (+ 409us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:22.034382 (+275816us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:22.034432 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:22.049669 (+ 15237us) log.cc:842] Serialized 26 byte log entry
0808 01:11:22.312683 (+263014us) raft_consensus.cc:1631] Marking committed up to 9
0808 01:11:22.360669 (+ 47986us) log.cc:842] Serialized 14 byte log entry
0808 01:11:22.360962 (+ 293us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:22.361073 (+ 111us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:23.081330 (+720257us) spinlock_profiling.cc:243] Waited 3.99 ms on lock 0x7b180035f860. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349b17764 00007fe349b1ab11 00007fe349b1aac9 00007fe349b1aa59 00007fe349b1aa21 00007fe349b19d1d 00007fe352d7b204 00007fe352d7b039 00007fe349af9dc6
0808 01:11:23.287130 (+205800us) spinlock_profiling.cc:243] Waited 8.79 ms on lock 0x7b180035f860. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349b17638 00007fe349b1ab11 00007fe349b1aac9 00007fe349b1aa59 00007fe349b1aa21 00007fe349b19d1d 00007fe352d7b204 00007fe352d7b039 00007fe349af9dc6
0808 01:11:23.370683 (+ 83553us) raft_consensus.cc:1671] finished
0808 01:11:23.370715 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:23.370876 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47263,"prepare.queue_time_us":54985,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":59,"raft.queue_time_us":117706,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":250,"spinlock_wait_cycles":29393024,"thread_start_us":434055,"threads_started":2,"wal-append.queue_time_us":715064}
I20260808 01:11:23.682503 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:23.294468 1518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2125) took 2006 ms. Trace:
W20260808 01:11:23.595806 1489 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.000s sys 0.009s
W20260808 01:11:23.685238 1489 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.599s user 0.000s sys 0.011s
W20260808 01:11:23.686128 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 902445us
I20260808 01:11:23.684420 1518 rpcz_store.cc:276] 0808 01:11:21.288310 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:21.757095 (+468785us) service_pool.cc:227] Handling call
0808 01:11:21.757496 (+ 401us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:21.865143 (+107647us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:21.865183 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:21.865650 (+ 467us) log.cc:842] Serialized 26 byte log entry
0808 01:11:22.072093 (+206443us) raft_consensus.cc:1631] Marking committed up to 9
0808 01:11:22.481134 (+409041us) log.cc:842] Serialized 14 byte log entry
0808 01:11:22.515363 (+ 34229us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:23.294020 (+778657us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54000c7954. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:23.294064 (+ 44us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:23.294130 (+ 66us) raft_consensus.cc:1671] finished
0808 01:11:23.294240 (+ 110us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:23.294394 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38381,"prepare.queue_time_us":59062,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":50,"raft.queue_time_us":46137,"raft.run_cpu_time_us":348,"raft.run_wall_time_us":118168,"spinlock_wait_cycles":246881536,"thread_start_us":469195,"threads_started":2,"wal-append.queue_time_us":340700}
I20260808 01:11:23.596309 1517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2130) took 1870 ms. Trace:
I20260808 01:11:23.689858 1519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2126) took 2397 ms. Trace:
I20260808 01:11:23.688030 1517 rpcz_store.cc:276] 0808 01:11:21.725977 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:21.943620 (+217643us) service_pool.cc:227] Handling call
0808 01:11:21.952029 (+ 8409us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:21.967190 (+ 15161us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:21.967323 (+ 133us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:21.968266 (+ 943us) log.cc:842] Serialized 26 byte log entry
0808 01:11:22.245483 (+277217us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:22.261104 (+ 15621us) log.cc:842] Serialized 14 byte log entry
0808 01:11:22.261321 (+ 217us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:22.261393 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:23.595354 (+1333961us) raft_consensus.cc:1671] finished
0808 01:11:23.595466 (+ 112us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:23.596000 (+ 534us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4665,"prepare.queue_time_us":10845,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":17289,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":22950,"thread_start_us":149227,"threads_started":1,"wal-append.queue_time_us":310720}
I20260808 01:11:23.317137 1521 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:23.597182 1522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2240) took 2081 ms. Trace:
I20260808 01:11:23.692680 4935 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:23.690397 1519 rpcz_store.cc:276] 0808 01:11:21.292348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:21.757752 (+465404us) service_pool.cc:227] Handling call
0808 01:11:21.758058 (+ 306us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:21.769393 (+ 11335us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:21.769423 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:21.769856 (+ 433us) log.cc:842] Serialized 26 byte log entry
0808 01:11:22.027104 (+257248us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:22.076844 (+ 49740us) log.cc:842] Serialized 14 byte log entry
0808 01:11:22.083461 (+ 6617us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:22.083559 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:22.651215 (+567656us) spinlock_profiling.cc:243] Waited 9.45 ms on lock 0x7b18001f2160. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349b17764 00007fe349b1ab11 00007fe349b1aac9 00007fe349b1aa59 00007fe349b1aa21 00007fe349b19d1d 00007fe352d7b204 00007fe352d7b039 00007fe349af9dc6
0808 01:11:23.293715 (+642500us) spinlock_profiling.cc:243] Waited 78.6 ms on lock 0x7b18001f2160. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349b17799 00007fe349b1ab11 00007fe349b1aac9 00007fe349b1aa59 00007fe349b1aa21 00007fe349b19d1d 00007fe352d7b204 00007fe352d7b039 00007fe349af9dc6
0808 01:11:23.676071 (+382356us) spinlock_profiling.cc:243] Waited 9.65 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:23.689361 (+ 13290us) raft_consensus.cc:1671] finished
0808 01:11:23.689392 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:23.689635 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":123615,"prepare.queue_time_us":49435,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":3810,"raft.queue_time_us":128391,"raft.run_cpu_time_us":1747,"raft.run_wall_time_us":27658,"spinlock_wait_cycles":224684032,"thread_start_us":197704,"threads_started":1,"wal-append.queue_time_us":795775}
W20260808 01:11:23.409942 1523 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58280 (request call id 2062) took 5517 ms (client timeout 1652 ms). Trace:
I20260808 01:11:23.317749 1524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58280 (request call id 2134) took 1332 ms. Trace:
I20260808 01:11:23.715297 1524 rpcz_store.cc:276] 0808 01:11:21.985202 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.161295 (+1176093us) service_pool.cc:227] Handling call
0808 01:11:23.317666 (+156371us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:23.716601 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:23.716935 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:23.717720 1519 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58266 (request call id 2246) took 1468 ms (client timeout 1806 ms). Trace:
W20260808 01:11:23.718001 1524 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36560 (request call id 667) took 1311 ms (client timeout 1717 ms). Trace:
W20260808 01:11:23.718159 1519 rpcz_store.cc:269] 0808 01:11:22.249085 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.695824 (+1446739us) service_pool.cc:227] Handling call
0808 01:11:23.717628 (+ 21804us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:23.718657 1524 rpcz_store.cc:269] 0808 01:11:22.406690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.716465 (+1309775us) service_pool.cc:227] Handling call
0808 01:11:23.717918 (+ 1453us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:23.400568 1525 rpcz_store.cc:276] 0808 01:11:21.286598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:21.436944 (+150346us) service_pool.cc:227] Handling call
0808 01:11:21.470713 (+ 33769us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:21.516038 (+ 45325us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:21.516080 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:21.516779 (+ 699us) log.cc:842] Serialized 26 byte log entry
0808 01:11:21.959170 (+442391us) raft_consensus.cc:1631] Marking committed up to 9
0808 01:11:21.993187 (+ 34017us) log.cc:842] Serialized 14 byte log entry
0808 01:11:22.054951 (+ 61764us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:22.055041 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:23.283572 (+1228531us) raft_consensus.cc:1671] finished
0808 01:11:23.283593 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:23.283753 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":167427,"prepare.queue_time_us":34592,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":53,"raft.queue_time_us":302882,"raft.run_cpu_time_us":401,"raft.run_wall_time_us":54960,"thread_start_us":441746,"threads_started":1,"wal-append.queue_time_us":880709}
I20260808 01:11:23.400806 1594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2636) took 1428 ms. Trace:
W20260808 01:11:23.608919 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 456103us
I20260808 01:11:23.720758 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:23.721880 1525 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:23.722455 1525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58266 (request call id 2255) took 1182 ms. Trace:
I20260808 01:11:23.722824 1525 rpcz_store.cc:276] 0808 01:11:22.540421 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.720028 (+1179607us) service_pool.cc:227] Handling call
0808 01:11:23.722369 (+ 2341us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:23.408995 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 270551us
W20260808 01:11:23.724125 1523 rpcz_store.cc:269] 0808 01:11:17.892322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:17.892453 (+ 131us) service_pool.cc:227] Handling call
0808 01:11:23.409642 (+5517189us) spinlock_profiling.cc:243] Waited 2.67 s on lock 0x7b540035d9d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:23.409807 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6136992128}
I20260808 01:11:23.412146 1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2349) took 1791 ms. Trace:
I20260808 01:11:23.692339 1522 rpcz_store.cc:276] 0808 01:11:21.516022 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:21.782595 (+266573us) service_pool.cc:227] Handling call
0808 01:11:21.832376 (+ 49781us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:21.904942 (+ 72566us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:21.904992 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 01:11:21.949815 (+ 44823us) log.cc:842] Serialized 52 byte log entry
0808 01:11:22.516580 (+566765us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:22.531746 (+ 15166us) log.cc:842] Serialized 14 byte log entry
0808 01:11:23.097209 (+565463us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:23.097279 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:23.281375 (+184096us) spinlock_profiling.cc:243] Waited 3.76 ms on lock 0x7b4800250758. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3518e67f1 00007fe3518dd8e7 00007fe3518dd77a 00007fe3519d195f 00007fe3519d1627 00007fe3519e5b2c 00007fe3519e5ad8 00007fe3519e5a58 00007fe3519e5a13 00007fe3519e4cbc 00007fe350966be6
0808 01:11:23.409616 (+128241us) spinlock_profiling.cc:243] Waited 3.37 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:23.596471 (+186855us) raft_consensus.cc:1671] finished
0808 01:11:23.596511 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:23.596912 (+ 401us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":169716,"prepare.queue_time_us":477839,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":59,"raft.queue_time_us":312679,"raft.run_cpu_time_us":3454,"raft.run_wall_time_us":1469033,"spinlock_wait_cycles":16398080,"thread_start_us":1085720,"threads_started":2,"wal-append.queue_time_us":1172931}
I20260808 01:11:23.727397 1601 rpcz_store.cc:276] 0808 01:11:21.620385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:21.634660 (+ 14275us) service_pool.cc:227] Handling call
0808 01:11:23.411950 (+1777290us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b5400304654. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:23.412076 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3957465344}
I20260808 01:11:23.614228 2202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 998) took 1414 ms. Trace:
W20260808 01:11:23.614984 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 350641us
I20260808 01:11:23.728504 2202 rpcz_store.cc:276] 0808 01:11:22.199575 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:22.269374 (+ 69799us) service_pool.cc:227] Handling call
0808 01:11:22.304384 (+ 35010us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:22.304818 (+ 434us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:22.304849 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:22.305400 (+ 551us) log.cc:842] Serialized 26 byte log entry
0808 01:11:22.400700 (+ 95300us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:22.400741 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:22.400801 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:23.613571 (+1212770us) raft_consensus.cc:1671] finished
0808 01:11:23.613622 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:23.613954 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7251,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":5423,"thread_start_us":94819,"threads_started":1,"wal-append.queue_time_us":600426}
W20260808 01:11:23.733145 2171 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 1.848s (SENT) [suppressed 20 similar messages]
W20260808 01:11:23.452180 2208 rpcz_store.cc:269] 0808 01:11:19.714616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:19.714758 (+ 142us) service_pool.cc:227] Handling call
0808 01:11:23.284747 (+3569989us) spinlock_profiling.cc:243] Waited 2.56 s on lock 0x7b54004c54d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:23.284902 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5889120128}
I20260808 01:11:23.430352 4553 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:23.736984 1567 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:23.621487 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1134) took 2254 ms. Trace:
I20260808 01:11:23.678292 1520 rpcz_store.cc:276] 0808 01:11:20.407718 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:20.435097 (+ 27379us) service_pool.cc:227] Handling call
0808 01:11:20.435653 (+ 556us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:20.436104 (+ 451us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:20.436143 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:20.436690 (+ 547us) log.cc:842] Serialized 26 byte log entry
0808 01:11:21.249074 (+812384us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:21.713559 (+464485us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b1800439480. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349ae4f96 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349b17799 00007fe349b1ab11
0808 01:11:21.786323 (+ 72764us) spinlock_profiling.cc:243] Waited 6.11 ms on lock 0x7b1800439480. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349b17799 00007fe349b1ab11 00007fe349b1aac9 00007fe349b1aa59 00007fe349b1aa21 00007fe349b19d1d 00007fe352d7b204 00007fe352d7b039 00007fe349af9dc6
0808 01:11:23.072091 (+1285768us) spinlock_profiling.cc:243] Waited 7.05 ms on lock 0x7b1800439480. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe34c6ce82c 00007fe349af99dc 00007fe352930a39 00007fe349b168bb 00007fe349b14cbf 00007fe349b14155 00007fe350414167 00007fe34ca49807 00007fe34ca58f0f
0808 01:11:23.072320 (+ 229us) log.cc:842] Serialized 14 byte log entry
0808 01:11:23.089360 (+ 17040us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:23.089479 (+ 119us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:23.295337 (+205858us) spinlock_profiling.cc:243] Waited 6.33 ms on lock 0x7b1800439480. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349ae4f96 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511
0808 01:11:23.296920 (+ 1583us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:23.654842 (+357922us) raft_consensus.cc:1671] finished
0808 01:11:23.657348 (+ 2506us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:23.677494 (+ 20146us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":264094,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":42603,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":5923,"spinlock_wait_cycles":96230912,"thread_start_us":1135188,"threads_started":2,"wal-append.queue_time_us":1014505}
I20260808 01:11:23.621290 4293 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:23.420046 4295 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:23.622243 4593 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:23.625317 4651 consensus_queue.cc:237] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.614133 1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 636) took 1621 ms. Trace:
I20260808 01:11:23.431058 4688 consensus_queue.cc:237] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.629495 4690 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:23.631583 4713 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:23.634352 4760 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:23.636759 4821 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:23.435823 4844 raft_consensus.cc:729] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:23.640379 4886 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:23.289620 4887 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:23.540833 4910 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:23.648569 4891 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:23.649796 4895 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:23.545111 4922 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:23.539114 4934 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:23.636868 4811 consensus_meta.cc:287] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.526s user 0.006s sys 0.012s
I20260808 01:11:23.546149 4936 consensus_queue.cc:237] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.653651 4905 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:23.693765 4935 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:23.653592 4896 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:23.671624 1445 rpcz_store.cc:276] 0808 01:11:21.541405 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:21.541735 (+ 330us) service_pool.cc:227] Handling call
0808 01:11:21.542132 (+ 397us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:21.811325 (+269193us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:21.811372 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:21.811426 (+ 54us) raft_consensus.cc:1631] Marking committed up to 9
0808 01:11:21.811477 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:23.286187 (+1474710us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b54001d2254. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:23.286236 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:23.286350 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":618536192}
I20260808 01:11:23.691578 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:23.652846 1408 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:23.683719 1516 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:23.701064 4926 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:23.701686 4953 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:23.704694 4757 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:23.720238 1594 rpcz_store.cc:276] 0808 01:11:21.972596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:21.972729 (+ 133us) service_pool.cc:227] Handling call
0808 01:11:22.034043 (+ 61314us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:23.283080 (+1249037us) spinlock_profiling.cc:243] Waited 39.9 ms on lock 0x7b540016b0a0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34c9cf730 00007fe34ca41300 00007fe34ca42371 00007fe34ca3eedf 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc
0808 01:11:23.400521 (+117441us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:23.400743 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":91881856}
W20260808 01:11:23.757469 1516 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58266 (request call id 2243) took 1521 ms (client timeout 1797 ms). Trace:
I20260808 01:11:23.755527 1600 rpcz_store.cc:276] 0808 01:11:21.992256 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:22.074428 (+ 82172us) service_pool.cc:227] Handling call
0808 01:11:22.074697 (+ 269us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:22.081408 (+ 6711us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:22.081442 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 01:11:22.082063 (+ 621us) log.cc:842] Serialized 52 byte log entry
0808 01:11:23.424102 (+1342039us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:23.611152 (+187050us) log.cc:842] Serialized 14 byte log entry
0808 01:11:23.612647 (+ 1495us) log.cc:842] Serialized 14 byte log entry
0808 01:11:23.613095 (+ 448us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:23.613310 (+ 215us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:23.613439 (+ 129us) raft_consensus.cc:1671] finished
0808 01:11:23.613472 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:23.613785 (+ 313us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9050,"prepare.queue_time_us":323520,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":1508,"raft.queue_time_us":100581,"raft.run_cpu_time_us":710,"raft.run_wall_time_us":24234,"thread_start_us":1289123,"threads_started":1,"wal-append.queue_time_us":1149671}
I20260808 01:11:23.765244 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:23.766228 1525 raft_consensus.cc:2454] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
W20260808 01:11:23.766821 1525 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36560 (request call id 663) took 1471 ms (client timeout 1945 ms). Trace:
I20260808 01:11:23.767061 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:23.768384 1518 raft_consensus.cc:2454] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:23.769186 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:23.769342 1518 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58266 (request call id 2254) took 1313 ms (client timeout 1578 ms). Trace:
I20260808 01:11:23.770092 4934 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:23.771059 4811 raft_consensus.cc:2726] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:23.772037 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:23.772874 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 121348us
I20260808 01:11:23.773114 1523 raft_consensus.cc:2454] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:23.773896 1523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58266 (request call id 2247) took 1514 ms. Trace:
I20260808 01:11:23.774287 1523 rpcz_store.cc:276] 0808 01:11:22.259965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.725296 (+1465331us) service_pool.cc:227] Handling call
0808 01:11:23.771772 (+ 46476us) spinlock_profiling.cc:243] Waited 30.5 ms on lock 0x7b5800035d98. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adf851 00007fe349adf702 00007fe3529cfb09 00007fe35190b730 00007fe3519d179b 00007fe3519d1821 00007fe3519811b0 00007fe351980a94 00007fe34b068c04 00007fe34b068b92
0808 01:11:23.773841 (+ 2069us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":70227200}
I20260808 01:11:23.770116 1520 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:23.775943 1520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58266 (request call id 2248) took 1512 ms. Trace:
I20260808 01:11:23.776938 1520 rpcz_store.cc:276] 0808 01:11:22.263232 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.740183 (+1476951us) service_pool.cc:227] Handling call
0808 01:11:23.768915 (+ 28732us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b5800035d98. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adf851 00007fe349adf702 00007fe3529cfb09 00007fe35190b730 00007fe3519d179b 00007fe3519d1821 00007fe3519811b0 00007fe351980a94 00007fe34b068c04 00007fe34b068b92
0808 01:11:23.775884 (+ 6969us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36196736}
W20260808 01:11:23.777318 1517 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58280 (request call id 2142) took 1787 ms (client timeout 2062 ms). Trace:
W20260808 01:11:23.769892 1518 rpcz_store.cc:269] 0808 01:11:22.456030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.687720 (+1231690us) service_pool.cc:227] Handling call
0808 01:11:23.766878 (+ 79158us) spinlock_profiling.cc:243] Waited 29.7 ms on lock 0x7b5800035d98. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adf851 00007fe349adfbce 000000000034aea9 00007fe35190ba30 00007fe3519d17b4 00007fe3519d1821 00007fe3519811b0 00007fe351980a94 00007fe34b068c04 00007fe34b068b92
0808 01:11:23.769267 (+ 2389us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":68240128}
W20260808 01:11:23.777943 1517 rpcz_store.cc:269] 0808 01:11:21.989675 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.691041 (+1701366us) service_pool.cc:227] Handling call
0808 01:11:23.777253 (+ 86212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:23.771116 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:23.775585 4896 raft_consensus.cc:2726] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:11:23.780161 4922 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:23.780682 4926 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:23.761173 1516 rpcz_store.cc:269] 0808 01:11:22.234260 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.681029 (+1446769us) service_pool.cc:227] Handling call
0808 01:11:23.755110 (+ 74081us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:23.783707 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:23.784795 1522 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:23.785418 1522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58266 (request call id 2250) took 1462 ms. Trace:
I20260808 01:11:23.785820 1522 rpcz_store.cc:276] 0808 01:11:22.323387 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.728018 (+1404631us) service_pool.cc:227] Handling call
0808 01:11:23.783413 (+ 55395us) spinlock_profiling.cc:243] Waited 27.9 ms on lock 0x7b5800035d98. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adf851 00007fe349adf702 00007fe3529cfb09 00007fe35190b730 00007fe3519d179b 00007fe3519d1821 00007fe3519811b0 00007fe351980a94 00007fe34b068c04 00007fe34b068b92
0808 01:11:23.785335 (+ 1922us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":64223616}
W20260808 01:11:23.767241 1525 rpcz_store.cc:269] 0808 01:11:22.295564 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.723585 (+1428021us) service_pool.cc:227] Handling call
0808 01:11:23.766731 (+ 43146us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:23.790382 1519 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36560 (request call id 660) took 1495 ms (client timeout 1884 ms). Trace:
W20260808 01:11:23.790849 1519 rpcz_store.cc:269] 0808 01:11:22.294771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.719464 (+1424693us) service_pool.cc:227] Handling call
0808 01:11:23.770825 (+ 51361us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b5800035d98. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349adf851 00007fe349adf702 00007fe3529cfb09 00007fe35190b730 00007fe3519d179b 00007fe3519d1821 00007fe3519811b0 00007fe351980a94 00007fe34b068c04 00007fe34b068b92
0808 01:11:23.790285 (+ 19460us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35001088}
I20260808 01:11:23.791318 4844 consensus_queue.cc:260] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.792868 4844 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:23.794482 4887 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:23.742723 4958 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20260808 01:11:23.798557 4891 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.763245 4953 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.804080 4593 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:23.805259 4593 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.739308 2204 rpcz_store.cc:276] 0808 01:11:21.366870 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:21.371576 (+ 4706us) service_pool.cc:227] Handling call
0808 01:11:21.371868 (+ 292us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:21.385011 (+ 13143us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:21.385036 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:21.395876 (+ 10840us) log.cc:842] Serialized 26 byte log entry
0808 01:11:21.812587 (+416711us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:21.812642 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:21.812731 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:23.301070 (+1488339us) spinlock_profiling.cc:243] Waited 9.37 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:23.620455 (+319385us) raft_consensus.cc:1671] finished
0808 01:11:23.620517 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:23.621169 (+ 652us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25914,"raft.queue_time_us":15592,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":9410,"spinlock_wait_cycles":21557120,"thread_start_us":327859,"threads_started":1,"wal-append.queue_time_us":648544}
I20260808 01:11:23.808017 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:23.812834 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:23.783740 4926 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:23.812214 2171 leader_election.cc:334] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093): Timed out: RequestConsensusVote RPC to 127.31.212.66:37093 timed out after 1.884s (SENT)
I20260808 01:11:23.814157 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:23.818620 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:23.826588 4953 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:23.827447 4843 consensus_queue.cc:1046] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260808 01:11:23.831102 4842 consensus_meta.cc:287] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 1.462s user 0.010s sys 0.019s
I20260808 01:11:23.831156 4811 raft_consensus.cc:3032] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:23.831761 4811 raft_consensus.cc:729] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
W20260808 01:11:23.832217 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 94768.6us
I20260808 01:11:23.832849 4811 consensus_queue.cc:260] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.834045 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:23.834283 4811 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:23.835486 1517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58266 (request call id 2244) took 1601 ms. Trace:
I20260808 01:11:23.840314 1517 rpcz_store.cc:276] 0808 01:11:22.234552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.833589 (+1599037us) service_pool.cc:227] Handling call
0808 01:11:23.835406 (+ 1817us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:23.833950 2208 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 10.
I20260808 01:11:23.817708 2171 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:23.891778 4891 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:23.892127 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:23.893250 1516 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:23.840058 4842 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.896646 4833 consensus_queue.cc:1046] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:11:23.898847 4842 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:23.900926 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:23.904366 4593 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:23.905476 4907 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:23.907189 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:23.908828 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:23.909041 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:23.909740 1525 raft_consensus.cc:2379] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 4: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:23.909500 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:23.910194 1525 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36560 (request call id 666) took 1611 ms (client timeout 2098 ms). Trace:
I20260808 01:11:23.910504 1440 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:23.871752 4935 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:23.896970 1517 raft_consensus.cc:1261] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20260808 01:11:23.995038 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:23.914819 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:24.000803 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 213128us
I20260808 01:11:24.039129 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:24.036463 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:24.040141 4912 consensus_meta.cc:287] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.642s user 0.017s sys 0.017s
I20260808 01:11:24.040517 1520 raft_consensus.cc:2379] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:24.041656 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:24.042430 4907 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:24.042958 4733 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:24.043725 4733 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.044305 4842 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:24.045075 4842 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.046244 4733 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:24.047122 4733 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:24.047755 4733 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.049582 4652 raft_consensus.cc:1013] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: attempt to promote peer 95b661b03f1d460b9657e61d4b0247cd: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260808 01:11:24.050150 1516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2132) took 2323 ms. Trace:
I20260808 01:11:24.050536 1516 rpcz_store.cc:276] 0808 01:11:21.726750 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.895229 (+2168479us) service_pool.cc:227] Handling call
0808 01:11:24.035777 (+140548us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:24.049735 (+ 13958us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:24.049766 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:24.049830 (+ 64us) raft_consensus.cc:1631] Marking committed up to 2
0808 01:11:24.049871 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.049946 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:24.050054 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:24.051451 4733 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:24.052388 4833 raft_consensus.cc:3032] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:11:24.052865 4269 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
W20260808 01:11:24.053352 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 316755us
I20260808 01:11:24.053666 4269 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.053308 4960 raft_consensus.cc:2726] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:11:24.052395 1519 raft_consensus.cc:2379] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:24.056072 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:24.056998 4269 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:24.052912 4833 raft_consensus.cc:729] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:24.058142 1519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2133) took 2073 ms. Trace:
I20260808 01:11:24.058561 1519 rpcz_store.cc:276] 0808 01:11:21.984831 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.057196 (+2072365us) service_pool.cc:227] Handling call
0808 01:11:24.057573 (+ 377us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:24.057799 (+ 226us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:24.057824 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:24.057885 (+ 61us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:24.057917 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.057980 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:24.058077 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:24.002607 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:24.056147 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:23.917069 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:23.910643 1525 rpcz_store.cc:269] 0808 01:11:22.299234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.908365 (+1609131us) service_pool.cc:227] Handling call
0808 01:11:23.910140 (+ 1775us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:24.060663 1517 pending_rounds.cc:76] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5: Aborting all ops after (but not including) 5
I20260808 01:11:24.030807 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:23.915007 1521 raft_consensus.cc:2454] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:24.059863 1522 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:24.040941 4912 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.064286 1522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36560 (request call id 669) took 1354 ms. Trace:
I20260808 01:11:24.066787 1522 rpcz_store.cc:276] 0808 01:11:22.710123 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.936712 (+1226589us) service_pool.cc:227] Handling call
0808 01:11:24.064234 (+127522us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:24.058636 4833 consensus_queue.cc:260] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.070195 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:24.071269 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.072494 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.072975 1596 raft_consensus.cc:2454] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:11:24.074757 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.060949 1517 pending_rounds.cc:97] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 5.6
I20260808 01:11:24.001349 4843 raft_consensus.cc:2726] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:11:24.082561 1521 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58266 (request call id 2242) took 1842 ms (client timeout 1771 ms). Trace:
I20260808 01:11:23.912312 4731 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:24.093819 1523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36560 (request call id 665) took 1796 ms. Trace:
I20260808 01:11:23.915956 4945 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:24.083113 1521 rpcz_store.cc:269] 0808 01:11:22.233928 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.186109 (+952181us) service_pool.cc:227] Handling call
0808 01:11:24.066111 (+880002us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:24.079468 4935 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:24.090977 1567 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:24.092097 4842 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:24.084231 1518 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:24.095727 1523 rpcz_store.cc:276] 0808 01:11:22.297772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.862377 (+1564605us) service_pool.cc:227] Handling call
0808 01:11:24.093742 (+231365us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:24.106935 1518 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58266 (request call id 2251) took 1777 ms (client timeout 2185 ms). Trace:
I20260808 01:11:24.097524 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:24.098059 1517 raft_consensus.cc:2894] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Aborting config change with OpId 5.6: Aborted: Op aborted by new leader
I20260808 01:11:24.099588 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:23.925690 1338 catalog_manager.cc:5495] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 5 to 8, leader changed from 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 8 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:11:24.107390 1518 rpcz_store.cc:269] 0808 01:11:22.329413 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.873753 (+1544340us) service_pool.cc:227] Handling call
0808 01:11:24.106873 (+233120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:23.935602 4553 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.108480 1524 raft_consensus.cc:2454] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:24.094447 4731 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:24.112244 1517 raft_consensus.cc:2866] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:11:24.112370 1524 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36560 (request call id 668) took 1575 ms (client timeout 1850 ms). Trace:
I20260808 01:11:24.112993 4664 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:24.113981 1516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:58250 (request call id 1087) took 2160 ms. Trace:
I20260808 01:11:24.132200 1516 rpcz_store.cc:276] 0808 01:11:21.953840 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.051564 (+2097724us) service_pool.cc:227] Handling call
0808 01:11:24.113879 (+ 62315us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:24.114164 4935 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:24.116086 4833 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:24.096113 4922 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:24.117201 4593 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:24.139431 4922 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.142082 4935 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:24.142318 4922 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:24.143167 4907 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:24.144027 4907 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:24.146692 4907 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:24.118510 1339 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55758 (request call id 335) took 10760 ms. Trace:
W20260808 01:11:24.119172 4871 consensus_meta.cc:287] T c12c84f631ee4ca7b405e28317db1b8a P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.124s user 0.010s sys 0.018s
I20260808 01:11:24.120118 4688 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:24.120424 1447 raft_consensus.cc:1261] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260808 01:11:24.065737 4269 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:24.127061 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:24.125277 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:24.156486 1523 raft_consensus.cc:2379] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 7: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:24.130213 1519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2135) took 2144 ms. Trace:
I20260808 01:11:24.157239 1519 rpcz_store.cc:276] 0808 01:11:21.985554 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.065078 (+2079524us) service_pool.cc:227] Handling call
0808 01:11:24.106421 (+ 41343us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:24.128090 (+ 21669us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:24.128120 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:24.128165 (+ 45us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:24.129941 (+ 1776us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.130024 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:24.130144 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:24.132452 1524 rpcz_store.cc:269] 0808 01:11:22.536783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.916625 (+1379842us) service_pool.cc:227] Handling call
0808 01:11:24.112319 (+195694us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:24.154038 1447 pending_rounds.cc:76] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d: Aborting all ops after (but not including) 3
I20260808 01:11:24.159374 1447 pending_rounds.cc:97] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d: Aborting uncommitted NO_OP operation due to leader change: 4.4
W20260808 01:11:24.160974 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 70237.7us
I20260808 01:11:24.136691 1521 raft_consensus.cc:1226] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:24.139236 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:24.153182 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.159698 1447 raft_consensus.cc:2866] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:24.169070 1521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2136) took 2183 ms. Trace:
I20260808 01:11:24.169462 1521 rpcz_store.cc:276] 0808 01:11:21.985871 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.110857 (+2124986us) service_pool.cc:227] Handling call
0808 01:11:24.129250 (+ 18393us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:24.168777 (+ 39527us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.169006 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:24.158653 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:24.176585 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:24.180752 1440 raft_consensus.cc:1261] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260808 01:11:24.185599 1521 raft_consensus.cc:2454] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 3.
I20260808 01:11:24.181350 1440 pending_rounds.cc:76] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d: Aborting all ops after (but not including) 5
I20260808 01:11:24.188400 1440 pending_rounds.cc:97] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260808 01:11:24.188666 1440 raft_consensus.cc:2866] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:24.197109 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:24.197939 1524 raft_consensus.cc:2454] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:24.160806 4596 raft_consensus.cc:2726] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:24.192054 1525 raft_consensus.cc:2454] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 3.
I20260808 01:11:24.133100 4945 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.205902 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:24.139240 4593 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.212528 1442 raft_consensus.cc:1226] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 7. Current term is 8. Ops: []
I20260808 01:11:24.191913 1338 catalog_manager.cc:5495] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:11:24.220135 1517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2131) took 2490 ms. Trace:
I20260808 01:11:24.223347 1517 rpcz_store.cc:276] 0808 01:11:21.726373 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:23.881858 (+2155485us) service_pool.cc:227] Handling call
0808 01:11:23.896463 (+ 14605us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:24.216702 (+320239us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.216972 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:24.217510 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:24.226931 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:24.225929 1519 raft_consensus.cc:2454] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:24.155164 4688 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:24.233148 1443 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 10.
I20260808 01:11:24.243652 1523 raft_consensus.cc:2454] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:24.245061 1523 raft_consensus.cc:1226] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:24.245664 1523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2249) took 1938 ms. Trace:
I20260808 01:11:24.245980 1523 rpcz_store.cc:276] 0808 01:11:22.307144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.244486 (+1937342us) service_pool.cc:227] Handling call
0808 01:11:24.244898 (+ 412us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:24.245445 (+ 547us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.245597 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:24.247556 1523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2144) took 1859 ms. Trace:
I20260808 01:11:24.161213 4664 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.248107 4945 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:24.224380 4293 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:24.261448 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.263461 1523 rpcz_store.cc:276] 0808 01:11:22.388437 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.246729 (+1858292us) service_pool.cc:227] Handling call
0808 01:11:24.247049 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:24.247245 (+ 196us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:24.247273 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:24.247340 (+ 67us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:24.247373 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.247419 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:24.247508 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:24.259090 4886 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:24.272841 4886 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:24.273896 1328 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet a83a963773094d6db6495b7009fd8690 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6b09dbefd24b4d52b385c4dff5697dc5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
W20260808 01:11:24.274865 1328 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 54388.7us
I20260808 01:11:24.275625 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.277125 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.247902 4935 consensus_queue.cc:1046] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260808 01:11:24.278868 4935 consensus_queue.cc:1225] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer a4862d0938c84fc4bd840e4c17acae78 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260808 01:11:24.278877 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.279615 4935 raft_consensus.cc:3032] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:11:24.279968 4935 raft_consensus.cc:729] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:24.280395 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.281193 1439 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:24.281103 4935 consensus_queue.cc:260] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.282256 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.283416 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.284466 1520 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:24.278388 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.258492 1521 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 9.
I20260808 01:11:24.216760 4269 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.292395 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.292718 1440 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 9.
I20260808 01:11:24.222332 4887 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.296499 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.297034 4731 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:24.298126 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.298748 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.299019 4664 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:24.299432 1596 raft_consensus.cc:2454] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:24.299877 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.300520 1443 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 9.
I20260808 01:11:24.299355 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.303720 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.306689 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.295047 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:24.306627 1440 raft_consensus.cc:2454] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:24.309763 1448 raft_consensus.cc:2396] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260808 01:11:24.311821 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.318251 1443 raft_consensus.cc:2379] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:24.318630 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.147665 1339 rpcz_store.cc:276] 0808 01:11:13.358678 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:13.360089 (+ 1411us) service_pool.cc:227] Handling call
0808 01:11:24.118188 (+10758099us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0808 01:11:18.894718 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:11:18.895396 (+ 678us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:11:18.895466 (+ 70us) write_op.cc:435] Acquired schema lock
0808 01:11:18.895482 (+ 16us) tablet.cc:660] Decoding operations
0808 01:11:19.416955 (+521473us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:11:19.417048 (+ 93us) write_op.cc:641] Partition lock acquired for write op
0808 01:11:19.417074 (+ 26us) tablet.cc:683] Acquiring locks for 17 operations
0808 01:11:19.478610 (+ 61536us) tablet.cc:699] Row locks acquired
0808 01:11:19.478649 (+ 39us) write_op.cc:260] PREPARE: finished
0808 01:11:19.478857 (+ 208us) write_op.cc:270] Start()
0808 01:11:19.479007 (+ 150us) write_op.cc:276] Timestamp: P: 1786151479478753 usec, L: 0
0808 01:11:19.479025 (+ 18us) op_driver.cc:331] REPLICATION: starting
0808 01:11:19.485785 (+ 6760us) log.cc:842] Serialized 6702 byte log entry
0808 01:11:20.389697 (+903912us) op_driver.cc:444] REPLICATION: finished
0808 01:11:21.180159 (+790462us) write_op.cc:301] APPLY: starting
0808 01:11:21.180287 (+ 128us) tablet.cc:1365] starting BulkCheckPresence
0808 01:11:21.181211 (+ 924us) tablet.cc:1368] finished BulkCheckPresence
0808 01:11:21.181495 (+ 284us) tablet.cc:1370] starting ApplyRowOperation cycle
0808 01:11:21.477082 (+295587us) tablet.cc:1381] finished ApplyRowOperation cycle
0808 01:11:21.619554 (+142472us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=17
0808 01:11:21.635262 (+ 15708us) write_op.cc:312] APPLY: finished
0808 01:11:21.714290 (+ 79028us) log.cc:842] Serialized 119 byte log entry
0808 01:11:21.949632 (+235342us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:11:21.967486 (+ 17854us) write_op.cc:454] Released schema lock
0808 01:11:21.976720 (+ 9234us) write_op.cc:341] FINISH: Updating metrics
0808 01:11:22.042688 (+ 65968us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:11:22.053746 (+ 11058us) write_op.cc:454] Released schema lock
0808 01:11:22.699254 (+645508us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
Metrics: {"reported_tablets":41,"child_traces":[["op",{"apply.queue_time_us":790006,"apply.run_cpu_time_us":16157,"apply.run_wall_time_us":918240,"num_ops":17,"prepare.queue_time_us":449106,"prepare.run_cpu_time_us":56378,"prepare.run_wall_time_us":840107,"raft.queue_time_us":405910,"raft.run_cpu_time_us":9392,"raft.run_wall_time_us":249720,"replication_time_us":910532,"spinlock_wait_cycles":24056704,"thread_start_us":1012408,"threads_started":5,"wal-append.queue_time_us":528816,"wal-append.run_cpu_time_us":20069,"wal-append.run_wall_time_us":1491715}]]}
I20260808 01:11:24.322467 4568 consensus_queue.cc:1046] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.074s
I20260808 01:11:24.323899 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.324717 1444 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:24.326116 1444 raft_consensus.cc:1261] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:11:24.326658 1516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2139) took 2338 ms. Trace:
I20260808 01:11:24.327099 1516 rpcz_store.cc:276] 0808 01:11:21.988650 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.188214 (+2199564us) service_pool.cc:227] Handling call
0808 01:11:24.286695 (+ 98481us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:24.311329 (+ 24634us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:24.311374 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:24.311440 (+ 66us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:24.311484 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.311569 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:24.326480 (+ 14911us) spinlock_profiling.cc:243] Waited 58.3 ms on lock 0x7b540037f0d0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:24.326592 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1413,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":15566,"spinlock_wait_cycles":134061824}
I20260808 01:11:24.328652 4651 raft_consensus.cc:3032] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:24.329196 4651 raft_consensus.cc:729] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:24.328994 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:24.329945 1516 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:24.330454 4651 consensus_queue.cc:260] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:24.331907 1438 raft_consensus.cc:2354] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 6. Current term is 7.
I20260808 01:11:24.335739 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.333360 1524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2241) took 2100 ms. Trace:
I20260808 01:11:24.338619 1524 rpcz_store.cc:276] 0808 01:11:22.233142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.223117 (+1989975us) service_pool.cc:227] Handling call
0808 01:11:24.260632 (+ 37515us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:24.313240 (+ 52608us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:24.313279 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:24.333003 (+ 19724us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:24.333073 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.333138 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:24.333277 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:24.341744 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.344404 1442 raft_consensus.cc:2396] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260808 01:11:24.348142 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:24.360724 2205 raft_consensus.cc:2396] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 10 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260808 01:11:24.345955 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:24.362973 1524 raft_consensus.cc:2454] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:24.346967 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:24.262096 4293 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:24.366689 1439 raft_consensus.cc:2396] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260808 01:11:24.370246 4936 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:24.370923 4936 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:24.371685 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:24.372044 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:24.372370 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:24.373049 4593 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:24.373090 2204 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:24.373835 1567 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:24.374835 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:24.375116 2171 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:24.375722 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:24.376137 4961 raft_consensus.cc:2781] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:24.376320 1603 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:24.376557 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.372437 2209 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:24.331604 4886 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:24.376586 4961 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:24.378222 4961 raft_consensus.cc:3037] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:24.378494 4981 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:24.379078 4887 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:24.375429 4731 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:24.367239 4935 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:11:24.379979 4977 log.cc:924] Time spent T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.116s user 0.001s sys 0.000s
I20260808 01:11:24.379863 4731 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.376680 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:24.381389 2204 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:24.332741 4553 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:24.375511 4295 consensus_meta.cc:287] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.505s user 0.015s sys 0.000s
I20260808 01:11:24.390910 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.379446 1338 catalog_manager.cc:5495] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:11:24.356935 4912 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:24.451215 1523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2253) took 1990 ms. Trace:
I20260808 01:11:24.451776 1523 rpcz_store.cc:276] 0808 01:11:22.404553 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.264903 (+1860350us) service_pool.cc:227] Handling call
0808 01:11:24.382017 (+117114us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:24.394383 (+ 12366us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:24.394428 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:24.394504 (+ 76us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:24.394554 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.394637 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:24.394781 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:24.388152 1447 raft_consensus.cc:1261] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:11:24.455390 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.456688 4731 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:24.373108 4936 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:11:24.464452 1489 leader_election.cc:334] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 1.925s (SENT)
I20260808 01:11:24.472817 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:24.496707 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 121731us
I20260808 01:11:24.497853 2171 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:24.498992 4593 raft_consensus.cc:2781] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:24.499531 4593 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:24.500034 4593 raft_consensus.cc:3037] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:24.524142 2171 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:24.533949 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.473179 4651 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:11:24.538991 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 77320.1us
I20260808 01:11:24.536861 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.542520 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.543885 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:24.521471 1338 catalog_manager.cc:5495] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 2 to 5, leader changed from 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 5 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:24.553144 1338 catalog_manager.cc:5495] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 2 to 5, leader changed from 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 5 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:24.554607 1338 catalog_manager.cc:5495] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 4 to 9, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 9 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:11:24.481830 4568 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:24.564288 4568 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.566488 4568 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:24.572467 1524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2256) took 1792 ms. Trace:
I20260808 01:11:24.450994 4295 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.572947 1524 rpcz_store.cc:276] 0808 01:11:22.780216 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.382738 (+1602522us) service_pool.cc:227] Handling call
0808 01:11:24.483936 (+101198us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:24.572052 (+ 88116us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:24.572091 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:24.572143 (+ 52us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:24.572188 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.572261 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:24.572402 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:24.575888 4733 raft_consensus.cc:3037] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:24.584515 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.591236 2204 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 8.
I20260808 01:11:24.573176 4553 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:24.597127 1516 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 8.
I20260808 01:11:24.597811 1603 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:24.599855 1408 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:24.600795 1445 raft_consensus.cc:1226] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:24.600839 1520 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 9.
I20260808 01:11:24.583387 4568 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:24.616639 4688 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:24.594312 4553 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.654271 1447 raft_consensus.cc:1261] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:11:24.655143 1338 catalog_manager.cc:5495] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 2 to 3, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 3 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:11:24.653537 4269 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:24.614481 4593 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.664330 1520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2148) took 1576 ms. Trace:
I20260808 01:11:24.673939 1520 rpcz_store.cc:276] 0808 01:11:23.087722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.663332 (+1575610us) service_pool.cc:227] Handling call
0808 01:11:24.663773 (+ 441us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:24.664033 (+ 260us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:24.664057 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:24.664100 (+ 43us) raft_consensus.cc:1631] Marking committed up to 8
0808 01:11:24.664126 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.664173 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:24.664275 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:24.674996 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 175206us
I20260808 01:11:24.587270 4961 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:24.623481 4886 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:11:24.670149 4936 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:24.685880 4936 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.686480 4593 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:24.689162 1521 raft_consensus.cc:1226] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 8. Current term is 9. Ops: [6.6-8.7]
I20260808 01:11:24.693979 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:24.659562 4981 raft_consensus.cc:2726] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:11:24.696177 1604 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.009s sys 0.073s
W20260808 01:11:24.710738 1604 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.637s user 0.010s sys 0.074s
I20260808 01:11:24.702684 2171 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:24.702706 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.680862 4664 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:24.653699 4568 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:24.687435 4295 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:24.730638 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.734380 1438 raft_consensus.cc:2379] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 9: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:24.731397 1598 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:24.717262 1603 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:24.704416 4912 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:24.745613 4961 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:24.747421 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.717061 4593 raft_consensus.cc:2781] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:24.743356 1338 catalog_manager.cc:5495] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 2 to 4, leader changed from 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 4 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:24.753942 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.761708 1523 raft_consensus.cc:2396] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260808 01:11:24.771656 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.770130 1604 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:24.746687 4953 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:24.756471 4593 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:24.793929 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.794345 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.777247 4596 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:24.801298 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 149054us
I20260808 01:11:24.797668 1338 catalog_manager.cc:5495] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:11:24.800921 2171 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:24.811451 2203 raft_consensus.cc:1261] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEARNER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260808 01:11:24.771085 4912 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.789113 4593 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:24.765342 4664 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:24.803231 4978 log.cc:924] Time spent T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
I20260808 01:11:24.821316 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:24.823719 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:24.820734 4935 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:24.832731 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.828841 1600 raft_consensus.cc:1226] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:24.830482 1603 raft_consensus.cc:1226] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:24.831197 1338 catalog_manager.cc:5495] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 3 to 4, leader changed from 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65) to 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68). New cstate: current_term: 4 leader_uuid: "95b661b03f1d460b9657e61d4b0247cd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:11:24.816444 4936 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:24.851094 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.853399 1602 raft_consensus.cc:2454] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:11:24.832432 4553 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:24.854512 2171 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:24.859882 4945 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:24.860512 4945 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.861093 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:24.861407 4664 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:24.861977 2201 raft_consensus.cc:2379] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 7: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
W20260808 01:11:24.862073 4833 consensus_meta.cc:287] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.713s user 0.013s sys 0.001s
I20260808 01:11:24.862777 4945 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:24.863253 4983 log.cc:924] Time spent T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260808 01:11:24.864167 4912 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:24.859341 4935 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:24.812582 4596 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.868292 1604 raft_consensus.cc:1261] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:11:24.800578 4953 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.868584 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.869915 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.870362 4953 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:24.870512 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:24.852864 4981 raft_consensus.cc:2781] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:24.871348 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:24.871884 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.872126 2204 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:24.872376 2202 raft_consensus.cc:1261] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260808 01:11:24.873023 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.873196 4733 raft_consensus.cc:2726] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:24.873426 32593 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260808 01:11:24.873961 4733 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:24.874212 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.874495 4733 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:24.871284 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:24.875732 1603 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:24.876573 1601 raft_consensus.cc:2396] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260808 01:11:24.876917 2205 raft_consensus.cc:1226] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:24.877229 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.878157 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.879069 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.879267 4935 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:24.879777 1597 raft_consensus.cc:2454] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 3.
I20260808 01:11:24.880882 1410 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:24.880546 1338 catalog_manager.cc:5495] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd reported cstate change: term changed from 4 to 5, leader changed from a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67) to 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65). New cstate: current_term: 5 leader_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:24.881340 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:24.882026 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 129676us
I20260808 01:11:24.882910 1410 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:24.883234 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 48160us
I20260808 01:11:24.884014 4295 raft_consensus.cc:2726] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:11:24.884732 4295 raft_consensus.cc:2726] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:24.885254 2209 raft_consensus.cc:1226] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:24.885666 4568 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:24.886430 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:24.886744 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:24.886837 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:24.888010 4664 raft_consensus.cc:2781] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:24.888176 2200 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260808 01:11:24.888509 4664 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:24.888969 4664 raft_consensus.cc:3037] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:11:24.889897 4985 log.cc:924] Time spent T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260808 01:11:24.871304 1598 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:24.876504 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.892665 2202 raft_consensus.cc:1261] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:11:24.893831 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:24.894622 2201 raft_consensus.cc:2379] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:24.896555 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.897354 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:24.896629 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:24.904487 4988 log.cc:924] Time spent T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
W20260808 01:11:24.907123 4984 log.cc:924] Time spent T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20260808 01:11:24.908289 1518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2137) took 2922 ms. Trace:
I20260808 01:11:24.908795 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:24.908692 1518 rpcz_store.cc:276] 0808 01:11:21.986182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.124068 (+2137886us) service_pool.cc:227] Handling call
0808 01:11:24.150853 (+ 26785us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:24.183056 (+ 32203us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:24.183105 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:24.212709 (+ 29604us) log.cc:842] Serialized 26 byte log entry
0808 01:11:24.613676 (+400967us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:24.823744 (+210068us) log.cc:842] Serialized 14 byte log entry
0808 01:11:24.825342 (+ 1598us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.825418 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:24.908040 (+ 82622us) raft_consensus.cc:1671] finished
0808 01:11:24.908074 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:24.908208 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":24871,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":4945,"thread_start_us":600991,"threads_started":2,"wal-append.queue_time_us":547937}
I20260808 01:11:24.910327 2201 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:24.913493 1410 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:24.908308 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.916895 1603 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:24.951323 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:24.934248 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:24.954440 2203 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:24.918037 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:24.954615 1518 raft_consensus.cc:3037] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:24.932366 4733 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:24.949306 4293 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:24.944331 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.977174 4886 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:24.977751 4886 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:11:24.952230 1517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36560 (request call id 661) took 2657 ms. Trace:
I20260808 01:11:24.966137 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:24.983917 1517 rpcz_store.cc:276] 0808 01:11:22.295074 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.233378 (+1938304us) service_pool.cc:227] Handling call
0808 01:11:24.272707 (+ 39329us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:24.273111 (+ 404us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:24.273136 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:24.273581 (+ 445us) log.cc:842] Serialized 26 byte log entry
0808 01:11:24.615476 (+341895us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:24.951672 (+336196us) log.cc:842] Serialized 14 byte log entry
0808 01:11:24.951938 (+ 266us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.952010 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:24.952051 (+ 41us) raft_consensus.cc:1671] finished
0808 01:11:24.952061 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:24.952176 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7494,"prepare.queue_time_us":321484,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":281,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":80,"spinlock_wait_cycles":2483456,"thread_start_us":628365,"threads_started":2,"wal-append.queue_time_us":458254}
I20260808 01:11:24.970373 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:24.986831 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:24.993114 1519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36560 (request call id 664) took 2697 ms. Trace:
I20260808 01:11:24.956281 1525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2245) took 2712 ms. Trace:
I20260808 01:11:24.994642 1518 raft_consensus.cc:2454] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:24.996279 2203 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:24.995985 2201 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:24.993753 1519 rpcz_store.cc:276] 0808 01:11:22.295820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.241587 (+1945767us) service_pool.cc:227] Handling call
0808 01:11:24.270244 (+ 28657us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:24.270907 (+ 663us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:24.270938 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:24.271610 (+ 672us) log.cc:842] Serialized 26 byte log entry
0808 01:11:24.617618 (+346008us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:24.992283 (+374665us) log.cc:842] Serialized 14 byte log entry
0808 01:11:24.992671 (+ 388us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.992765 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:24.992817 (+ 52us) raft_consensus.cc:1671] finished
0808 01:11:24.992833 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:24.992990 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46546,"prepare.queue_time_us":333825,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":320,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"thread_start_us":673215,"threads_started":2,"wal-append.queue_time_us":514677}
I20260808 01:11:24.998986 1567 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:24.987722 1517 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:25.009182 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:25.020555 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:25.021077 1521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2145) took 2566 ms. Trace:
I20260808 01:11:25.021456 1521 rpcz_store.cc:276] 0808 01:11:22.455160 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.291574 (+1836414us) service_pool.cc:227] Handling call
0808 01:11:24.664673 (+373099us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:24.699179 (+ 34506us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:25.020886 (+321707us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b54001a1b50. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:25.021016 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":788753664}
I20260808 01:11:25.025065 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.027608 1567 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:25.035167 1519 raft_consensus.cc:1226] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:25.036020 1519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36560 (request call id 670) took 1336 ms. Trace:
I20260808 01:11:25.036391 1519 rpcz_store.cc:276] 0808 01:11:23.709849 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:25.001468 (+1291619us) service_pool.cc:227] Handling call
0808 01:11:25.034938 (+ 33470us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:25.035707 (+ 769us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:25.035956 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:25.037091 1598 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
W20260808 01:11:25.026278 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 112893us
I20260808 01:11:24.997256 1525 rpcz_store.cc:276] 0808 01:11:22.243608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.219560 (+1975952us) service_pool.cc:227] Handling call
0808 01:11:24.290988 (+ 71428us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:24.351770 (+ 60782us) raft_consensus.cc:1520] Early marking committed up to index 7
0808 01:11:24.706849 (+355079us) log.cc:842] Serialized 14 byte log entry
0808 01:11:24.955945 (+249096us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:24.955990 (+ 45us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:24.956038 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.956097 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:24.956224 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":469749,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":87,"raft.queue_time_us":3774,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":25938,"thread_start_us":473984,"threads_started":2}
I20260808 01:11:25.048198 1521 raft_consensus.cc:1226] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:25.049185 1522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2143) took 2973 ms. Trace:
I20260808 01:11:25.049643 1522 rpcz_store.cc:276] 0808 01:11:22.076099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.193158 (+2117059us) service_pool.cc:227] Handling call
0808 01:11:24.234142 (+ 40984us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:24.260981 (+ 26839us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:24.261027 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:24.267352 (+ 6325us) log.cc:842] Serialized 26 byte log entry
0808 01:11:24.325495 (+ 58143us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b180047c980. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349af9345 00007fe352930a39 00007fe349b168bb 00007fe349b14cbf 00007fe349b16e22 00007fe34c6c05bb 00007fe34c6c6063 00007fe34c6c6411 00007fe34ca135cf
0808 01:11:24.608625 (+283130us) raft_consensus.cc:1631] Marking committed up to 3
0808 01:11:24.608677 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.608771 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:24.830115 (+221344us) raft_consensus.cc:1671] finished
0808 01:11:24.830140 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:25.048966 (+218826us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b540012f050. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:25.049115 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6344,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":4917,"spinlock_wait_cycles":48697856,"thread_start_us":282901,"threads_started":1,"wal-append.queue_time_us":477249}
I20260808 01:11:25.053400 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.057076 1601 raft_consensus.cc:2354] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 7. Current term is 8.
I20260808 01:11:25.057911 1525 raft_consensus.cc:1261] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260808 01:11:25.057204 1518 raft_consensus.cc:2454] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 4.
I20260808 01:11:25.065609 2205 raft_consensus.cc:1261] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:11:25.042748 4891 raft_consensus.cc:2781] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:25.068764 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:25.073215 2200 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:25.067543 1567 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:25.053951 4568 raft_consensus.cc:2781] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:25.091522 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.092549 1601 raft_consensus.cc:2354] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 8. Current term is 9.
W20260808 01:11:25.081247 4651 consensus_meta.cc:287] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.507s user 0.008s sys 0.004s
I20260808 01:11:25.096066 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:25.090946 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:25.098348 1517 raft_consensus.cc:1261] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260808 01:11:25.098459 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 159828us
I20260808 01:11:25.083674 4568 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:25.106647 1516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2147) took 2019 ms. Trace:
I20260808 01:11:25.105381 1521 raft_consensus.cc:1261] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:11:25.091192 4833 raft_consensus.cc:2781] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:25.088379 4891 raft_consensus.cc:686] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:25.110555 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.112782 1601 raft_consensus.cc:2362] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6. Re-sending same reply.
I20260808 01:11:25.114934 2205 raft_consensus.cc:2396] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:11:25.116923 4293 consensus_queue.cc:1035] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:11:25.118382 1522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58250 (request call id 1086) took 3211 ms. Trace:
I20260808 01:11:25.118800 1522 rpcz_store.cc:276] 0808 01:11:21.907719 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:25.105186 (+3197467us) service_pool.cc:227] Handling call
0808 01:11:25.118270 (+ 13084us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:25.121727 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.122584 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.123083 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.124172 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:25.124445 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.126085 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.123165 1598 raft_consensus.cc:2362] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6. Re-sending same reply.
I20260808 01:11:25.127137 1520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2149) took 1737 ms. Trace:
I20260808 01:11:25.108711 4568 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:25.127511 1520 rpcz_store.cc:276] 0808 01:11:23.390246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.675091 (+1284845us) service_pool.cc:227] Handling call
0808 01:11:24.710819 (+ 35728us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:24.717440 (+ 6621us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:24.717470 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:24.755813 (+ 38343us) log.cc:842] Serialized 564 byte log entry
0808 01:11:25.072134 (+316321us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:25.072183 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:25.072276 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:25.126958 (+ 54682us) raft_consensus.cc:1671] finished
0808 01:11:25.126974 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:25.127086 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":293051,"threads_started":1,"wal-append.queue_time_us":365212}
I20260808 01:11:25.128100 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.130421 1601 raft_consensus.cc:2396] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
W20260808 01:11:25.135187 4982 log.cc:924] Time spent T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.473s user 0.005s sys 0.001s
I20260808 01:11:25.140553 2172 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:25.142212 2172 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:25.143296 2172 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:25.145480 4733 raft_consensus.cc:2726] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:25.146638 4733 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:11:25.146973 4733 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:25.147282 4733 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:11:25.106056 4295 raft_consensus.cc:2781] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:25.148056 4945 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:11:25.148494 4945 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:25.148851 4945 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:11:25.149505 4295 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:25.149962 4295 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:25.153446 4596 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:25.157820 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:25.158255 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:25.158857 1520 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:25.161923 1594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2366) took 1096 ms. Trace:
I20260808 01:11:25.124821 1599 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:25.163901 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.172338 2200 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 7.
I20260808 01:11:25.111235 4833 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:25.213074 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:25.103683 4936 consensus_queue.cc:1035] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.078s
I20260808 01:11:25.215823 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.220476 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.140553 4981 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:25.162400 1594 rpcz_store.cc:276] 0808 01:11:24.065247 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.065432 (+ 185us) service_pool.cc:227] Handling call
0808 01:11:24.069170 (+ 3738us) raft_consensus.cc:1457] Updating replica for 3 ops
0808 01:11:24.070020 (+ 850us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:24.070049 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0808 01:11:24.070935 (+ 886us) log.cc:842] Serialized 78 byte log entry
0808 01:11:24.357750 (+286815us) raft_consensus.cc:1631] Marking committed up to 10
0808 01:11:24.367225 (+ 9475us) log.cc:842] Serialized 14 byte log entry
0808 01:11:24.381973 (+ 14748us) log.cc:842] Serialized 14 byte log entry
0808 01:11:24.459738 (+ 77765us) spinlock_profiling.cc:243] Waited 5.29 ms on lock 0x7b18003591a0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349a8e4ca 00007fe35316ff99 00007fe349952990 00007fe349b14864 00007fe349b14155 00007fe34ca373d0 00007fe34ca49576 00007fe34ca5a0d7 00007fe34ca5a058
0808 01:11:24.460174 (+ 436us) log.cc:842] Serialized 14 byte log entry
0808 01:11:24.460346 (+ 172us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:24.460500 (+ 154us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:24.661871 (+201371us) spinlock_profiling.cc:243] Waited 3.35 ms on lock 0x7b18003591a0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349b17638 00007fe349b1ab11 00007fe349b1aac9 00007fe349b1aa59 00007fe349b1aa21 00007fe349b19d1d 00007fe352d7b204 00007fe352d7b039 00007fe349af9dc6
0808 01:11:24.666124 (+ 4253us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b18003591a0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3529b68b1 00007fe3529c3661 00007fe349b17638 00007fe349b1ab11 00007fe349b1aac9 00007fe349b1aa59 00007fe349b1aa21 00007fe349b19d1d 00007fe352d7b204 00007fe352d7b039 00007fe349af9dc6
0808 01:11:25.134475 (+468351us) spinlock_profiling.cc:243] Waited 3.11 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:25.161694 (+ 27219us) raft_consensus.cc:1671] finished
0808 01:11:25.161719 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:25.161861 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2381,"prepare.queue_time_us":225008,"prepare.run_cpu_time_us":171,"prepare.run_wall_time_us":1809,"raft.queue_time_us":84986,"raft.run_cpu_time_us":421,"raft.run_wall_time_us":2771,"spinlock_wait_cycles":55625472,"thread_start_us":286315,"threads_started":1,"wal-append.queue_time_us":513553}
I20260808 01:11:25.234867 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.239143 4664 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.242039 4664 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:25.244700 4981 raft_consensus.cc:3037] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:25.235670 1596 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 7.
I20260808 01:11:25.218387 4833 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:25.258665 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.261899 1603 raft_consensus.cc:2354] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for earlier term 7. Current term is 8.
I20260808 01:11:25.248258 1565 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:25.263322 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 159759us
W20260808 01:11:25.265460 1490 leader_election.cc:334] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.532s (SENT)
I20260808 01:11:25.268594 1490 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:25.253034 1599 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 7.
I20260808 01:11:25.274950 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.271114 1565 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:25.276513 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 174901us
W20260808 01:11:25.276895 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 204269us
I20260808 01:11:25.284301 1408 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:25.286942 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:25.287595 1603 raft_consensus.cc:2379] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 5: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:25.288024 1599 raft_consensus.cc:2362] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6. Re-sending same reply.
I20260808 01:11:25.285756 4891 consensus_queue.cc:237] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:25.309629 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:25.313741 4664 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:25.314565 4664 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.315932 4961 raft_consensus.cc:2741] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260808 01:11:25.273425 4936 raft_consensus.cc:2741] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260808 01:11:25.321544 1599 raft_consensus.cc:3037] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 7
W20260808 01:11:25.318164 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 174917us
I20260808 01:11:25.285472 4935 raft_consensus.cc:2726] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:25.259655 4886 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:25.325582 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 57236.5us
I20260808 01:11:25.327888 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.337199 1599 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 7.
I20260808 01:11:25.327795 4664 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:25.326347 4688 consensus_queue.cc:1035] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.012s
I20260808 01:11:25.327996 5000 raft_consensus.cc:2781] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:25.339095 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.339290 5000 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:25.339686 5000 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:25.317072 4553 consensus_queue.cc:1035] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.023s
I20260808 01:11:25.340263 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.341032 1598 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:25.338420 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.107075 1516 rpcz_store.cc:276] 0808 01:11:23.087460 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.603542 (+1516082us) service_pool.cc:227] Handling call
0808 01:11:24.616120 (+ 12578us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:24.649573 (+ 33453us) raft_consensus.cc:1520] Early marking committed up to index 3
0808 01:11:24.954003 (+304430us) log.cc:842] Serialized 14 byte log entry
0808 01:11:25.106330 (+152327us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:25.106377 (+ 47us) raft_consensus.cc:1631] Marking committed up to 3
0808 01:11:25.106423 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:25.106492 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:25.106597 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15622,"prepare.queue_time_us":304651,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":19272,"raft.queue_time_us":3530,"thread_start_us":332291,"threads_started":2}
I20260808 01:11:25.343096 1487 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:25.343611 1410 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:25.268772 4953 consensus_queue.cc:1046] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260808 01:11:25.327283 1565 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:25.347410 2172 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
W20260808 01:11:25.347673 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 116189us
W20260808 01:11:25.344573 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 79162us
W20260808 01:11:25.332603 5006 log.cc:924] Time spent T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.171s user 0.000s sys 0.000s
I20260808 01:11:25.353099 4886 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 11 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:25.354094 1524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2146) took 2267 ms. Trace:
I20260808 01:11:25.354023 5015 consensus_queue.cc:1046] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260808 01:11:25.354599 1524 rpcz_store.cc:276] 0808 01:11:23.087103 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:24.583133 (+1496030us) service_pool.cc:227] Handling call
0808 01:11:24.619963 (+ 36830us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:24.670929 (+ 50966us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:24.670974 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:24.689896 (+ 18922us) log.cc:842] Serialized 26 byte log entry
0808 01:11:25.020824 (+330928us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:25.020897 (+ 73us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:25.020989 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:25.353868 (+332879us) raft_consensus.cc:1671] finished
0808 01:11:25.353893 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:25.354038 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9663,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":146,"thread_start_us":233556,"threads_started":1,"wal-append.queue_time_us":351222}
I20260808 01:11:25.355523 5015 raft_consensus.cc:3032] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:11:25.355962 5015 raft_consensus.cc:729] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:25.356302 4553 raft_consensus.cc:3032] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:25.356747 4553 raft_consensus.cc:729] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
W20260808 01:11:25.357204 4593 consensus_meta.cc:287] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.518s user 0.009s sys 0.001s
I20260808 01:11:25.356946 5015 consensus_queue.cc:260] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.357628 4593 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.357877 4553 consensus_queue.cc:260] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.358394 5015 raft_consensus.cc:3037] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:25.359180 4553 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:25.348667 4935 raft_consensus.cc:2781] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:25.276866 4945 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.360049 2200 raft_consensus.cc:1226] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:25.364513 1410 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:25.370843 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.366106 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:25.373018 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.371232 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.373384 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.368721 1596 raft_consensus.cc:1261] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:11:25.378674 1601 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 9.
I20260808 01:11:25.364848 5020 raft_consensus.cc:2781] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:25.379532 5020 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:25.379769 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:25.380693 1602 raft_consensus.cc:2379] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 7: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:25.380880 4886 raft_consensus.cc:2781] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:25.381249 1410 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:25.382673 1410 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:25.370918 4664 consensus_queue.cc:1035] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20260808 01:11:25.384105 1410 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:25.385164 5018 raft_consensus.cc:2781] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:25.385596 5018 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:25.385934 5018 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:25.360598 4935 raft_consensus.cc:686] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:25.379968 5020 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:25.387477 4935 consensus_queue.cc:237] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.389115 4887 raft_consensus.cc:2781] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:25.389523 4887 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:25.389884 4887 raft_consensus.cc:3037] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:25.378139 1603 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:25.381314 4886 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:25.370020 5023 raft_consensus.cc:2781] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:25.391667 4886 raft_consensus.cc:3037] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:25.395035 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:25.392248 5023 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:25.396206 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:25.397457 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:25.398329 4295 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:25.401000 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:25.401340 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:25.402829 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:25.404255 1525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2160) took 1020 ms. Trace:
I20260808 01:11:25.404862 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:25.444610 1438 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:25.405741 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:25.444991 1445 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:25.444731 1525 rpcz_store.cc:276] 0808 01:11:24.383874 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:25.074200 (+690326us) service_pool.cc:227] Handling call
0808 01:11:25.088354 (+ 14154us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:25.088638 (+ 284us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:25.088674 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:25.097450 (+ 8776us) log.cc:842] Serialized 26 byte log entry
0808 01:11:25.269736 (+172286us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:25.269774 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:25.270474 (+ 700us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:25.404070 (+133596us) raft_consensus.cc:1671] finished
0808 01:11:25.404089 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:25.404199 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":154749,"threads_started":1,"wal-append.queue_time_us":294877}
I20260808 01:11:25.445785 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:25.406876 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 79661.5us
I20260808 01:11:25.446569 1439 raft_consensus.cc:2396] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
W20260808 01:11:25.349033 2171 connection.cc:232] RPC call timeout handler was delayed by -1.0088255910086446s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3163s for a response
I20260808 01:11:25.442306 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:25.446729 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:25.442713 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 58685.5us
I20260808 01:11:25.447635 1524 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
W20260808 01:11:25.447745 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 104855us
I20260808 01:11:25.448457 1487 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:25.406085 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:25.407428 1489 leader_election.cc:334] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 1.573s (SENT)
W20260808 01:11:25.430967 1490 leader_election.cc:334] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.659s (SENT)
W20260808 01:11:25.450599 1490 leader_election.cc:334] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.676s (SENT)
I20260808 01:11:25.453240 1521 raft_consensus.cc:2379] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 9: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:25.457139 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:25.457970 1523 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 7.
I20260808 01:11:25.463467 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:25.465044 1410 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:25.465054 1489 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:25.469909 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:25.359800 4593 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:25.472468 1443 raft_consensus.cc:1226] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 7. Current term is 8. Ops: []
I20260808 01:11:25.448470 4961 raft_consensus.cc:2781] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:11:25.475755 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:25.452117 5019 raft_consensus.cc:2781] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:25.476677 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:25.483239 1444 raft_consensus.cc:2379] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:25.453598 5023 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:11:25.486230 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 78836us
I20260808 01:11:25.488554 1439 raft_consensus.cc:2379] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:25.464661 4688 raft_consensus.cc:2781] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:25.470260 4953 raft_consensus.cc:2726] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:25.378598 4568 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.478763 4757 raft_consensus.cc:2726] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:25.455796 4936 consensus_queue.cc:1035] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260808 01:11:25.509677 1597 raft_consensus.cc:2379] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 7: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
W20260808 01:11:25.518229 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 140507us
I20260808 01:11:25.443358 5000 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.505128 4295 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:25.482191 5019 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:25.470137 4960 consensus_queue.cc:1035] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260808 01:11:25.472398 4269 consensus_queue.cc:1035] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260808 01:11:25.514788 1438 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 8.
I20260808 01:11:25.511168 4568 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:25.511672 4733 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.507766 5018 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:25.594839 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 126832us
W20260808 01:11:25.603125 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 138150us
I20260808 01:11:25.678231 1410 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:25.719122 1442 raft_consensus.cc:1226] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:25.613111 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:25.634200 1522 raft_consensus.cc:2396] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260808 01:11:25.723455 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 345255us
I20260808 01:11:25.717056 2171 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:25.729674 2203 raft_consensus.cc:2379] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 11: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
W20260808 01:11:25.720542 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 229965us
I20260808 01:11:25.520660 4953 consensus_queue.cc:1035] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.197s
W20260808 01:11:25.730247 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 249814us
I20260808 01:11:25.622881 4945 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 10 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:25.536170 5019 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:25.689594 4596 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:25.739462 1410 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:25.797832 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 107010us
I20260808 01:11:25.717347 5029 raft_consensus.cc:2781] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:25.635039 4887 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:25.729604 4688 raft_consensus.cc:686] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:25.751570 1489 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:25.752516 4553 consensus_queue.cc:1046] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:11:25.753042 1599 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:11:25.763981 1568 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:25.771044 4945 raft_consensus.cc:2726] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:25.772356 4935 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:25.773605 4733 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 11 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:25.774341 4953 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:25.631649 4833 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:25.611724 5000 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:25.619176 4981 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.810969 4293 raft_consensus.cc:2781] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election won for term 5
W20260808 01:11:25.792034 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 51461.6us
I20260808 01:11:25.605837 4757 consensus_queue.cc:1046] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.190s
I20260808 01:11:25.811432 1599 raft_consensus.cc:595] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Received request to transfer leadership to TS 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:25.824599 1599 raft_consensus.cc:603] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:11:25.685706 5020 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:25.622160 5023 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:25.817960 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 93827us
W20260808 01:11:25.818676 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 67192us
I20260808 01:11:25.819355 1410 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:25.820096 1440 raft_consensus.cc:1226] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:25.802841 5029 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:25.829306 2200 raft_consensus.cc:2354] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 7. Current term is 8.
W20260808 01:11:25.829826 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 147965us
I20260808 01:11:25.831245 5023 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:25.831274 4733 consensus_queue.cc:1046] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:25.824558 4593 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:25.821915 4953 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.832304 5029 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:25.832580 4593 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.834057 4953 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:25.821715 5032 raft_consensus.cc:2781] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:25.834808 5032 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:25.834868 4833 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:25.835234 5032 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:25.835513 5030 consensus_queue.cc:1046] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:11:25.836237 4688 consensus_queue.cc:237] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.836932 5030 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:25.837527 5030 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.840066 4664 raft_consensus.cc:3032] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:11:25.840519 4664 raft_consensus.cc:729] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:25.840898 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:25.825976 4891 raft_consensus.cc:2726] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:25.841732 1442 raft_consensus.cc:2379] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 9: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:25.841374 4664 consensus_queue.cc:260] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:25.842058 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 95297.4us
I20260808 01:11:25.842515 5020 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:25.842764 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:25.843022 2205 raft_consensus.cc:1226] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:25.822384 4961 raft_consensus.cc:686] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 8 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:25.845784 4961 consensus_queue.cc:237] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.848908 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:25.849757 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:25.851516 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:25.842862 1568 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:25.821326 4935 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.852232 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:25.854444 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:25.855461 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:25.855568 1489 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:25.855907 5035 raft_consensus.cc:3032] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:25.856380 5000 raft_consensus.cc:2781] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260808 01:11:25.856508 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 44703.3us
I20260808 01:11:25.857105 5015 raft_consensus.cc:3032] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:25.874629 1445 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:25.810127 4596 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:25.877497 1447 raft_consensus.cc:2379] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 11: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:25.842674 4664 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:25.885995 1408 leader_election.cc:398] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Vote denied by peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571) with higher term. Message: Invalid argument: T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 7. Current term is 8.
I20260808 01:11:25.886349 1408 leader_election.cc:401] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260808 01:11:25.886693 1489 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:25.886962 1410 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:25.887979 1489 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:25.888052 1410 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:25.888873 1410 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:25.889487 4553 raft_consensus.cc:2726] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:25.892910 4922 raft_consensus.cc:2781] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:25.896173 4922 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:25.896530 4922 raft_consensus.cc:3037] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:25.897192 4295 raft_consensus.cc:2781] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:25.897619 4295 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:25.897976 4295 raft_consensus.cc:3037] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:25.880627 5015 raft_consensus.cc:729] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:25.900003 4871 raft_consensus.cc:2726] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:25.851116 5032 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:25.924199 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:25.884763 5013 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:25.926100 1410 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
W20260808 01:11:25.926632 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 46564.4us
I20260808 01:11:25.927294 4553 raft_consensus.cc:2726] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:11:25.876776 5035 raft_consensus.cc:729] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:25.880362 5000 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:25.925614 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 82889us
I20260808 01:11:25.935388 1447 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:25.935621 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:25.945360 1522 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:25.938745 2205 raft_consensus.cc:1226] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:25.935782 1445 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 8.
I20260808 01:11:25.944741 2171 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:25.878234 4953 consensus_queue.cc:1035] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.051s
I20260808 01:11:25.927218 5045 raft_consensus.cc:2726] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:25.953963 1568 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:25.958756 2172 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:25.974957 2172 leader_election.cc:302] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:25.927733 4757 consensus_queue.cc:1046] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260808 01:11:25.954202 5000 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:25.940521 5018 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:25.998245 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 53578.3us
I20260808 01:11:26.001856 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:25.953264 5004 raft_consensus.cc:2726] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571) with higher term. Message: Invalid argument: T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 7. Current term is 8.
I20260808 01:11:25.900692 5015 consensus_queue.cc:260] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:26.005414 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 51574.8us
I20260808 01:11:26.006254 1517 raft_consensus.cc:2379] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:25.970671 4833 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:26.000080 4293 raft_consensus.cc:686] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
W20260808 01:11:26.028501 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 140567us
I20260808 01:11:25.983601 5030 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:25.996729 4960 raft_consensus.cc:2726] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:25.981916 5042 raft_consensus.cc:2781] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:11:26.065959 1489 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:25.984191 4733 raft_consensus.cc:2781] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:25.976102 4886 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:25.947427 5013 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:26.068296 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 96008.2us
I20260808 01:11:26.029002 4891 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:26.086587 1517 raft_consensus.cc:2932] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 3 to 6, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:26.027593 5018 raft_consensus.cc:2781] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:26.092855 5015 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:26.067090 4887 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:26.108712 1489 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:26.086524 4733 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:11:26.100936 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 94353.3us
I20260808 01:11:26.114638 1489 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.110441 2171 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.028734 4833 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:26.116245 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 190174us
I20260808 01:11:26.103560 5042 raft_consensus.cc:686] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [term 8 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:26.121482 1568 leader_election.cc:398] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291) with higher term. Message: Invalid argument: T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 6. Current term is 7.
I20260808 01:11:26.121829 1568 leader_election.cc:401] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:11:26.117097 4733 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:26.119263 4269 consensus_queue.cc:1035] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260808 01:11:26.119717 4553 raft_consensus.cc:2781] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:11:26.149150 5054 raft_consensus.cc:3032] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:26.149398 4553 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:26.149677 1568 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:26.149614 5054 raft_consensus.cc:729] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:26.151028 1568 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:26.150784 5054 consensus_queue.cc:260] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:26.109143 5049 raft_consensus.cc:2726] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:26.137112 5048 raft_consensus.cc:3037] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:26.152094 5054 raft_consensus.cc:3037] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:26.152194 4731 raft_consensus.cc:2726] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:11:26.152851 4731 raft_consensus.cc:2726] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:26.138554 4960 raft_consensus.cc:2726] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291) with higher term. Message: Invalid argument: T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 6. Current term is 7.
I20260808 01:11:26.145052 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:26.148998 2171 leader_election.cc:398] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for earlier term 7. Current term is 8.
I20260808 01:11:26.138424 1489 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.149839 4553 raft_consensus.cc:3037] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:26.156229 1489 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.157217 4953 raft_consensus.cc:2726] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:26.142546 5013 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:26.157840 4953 raft_consensus.cc:2726] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:26.136983 5043 raft_consensus.cc:2781] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:26.159695 5019 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:26.160297 5043 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:26.160665 5043 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:26.154917 2171 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.161939 5019 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:26.161422 1489 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.163542 2171 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:26.140700 4886 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:26.164287 5049 raft_consensus.cc:2726] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:26.139777 4293 consensus_queue.cc:237] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:26.168967 5044 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:26.171890 5013 raft_consensus.cc:2781] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:26.126282 4935 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:26.144141 5032 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:26.118290 5038 consensus_queue.cc:1035] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
I20260808 01:11:26.182781 5029 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:26.180035 2171 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.180912 1568 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:26.124724 5030 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:26.104703 4891 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:26.184237 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 60507.7us
I20260808 01:11:26.202344 1568 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:26.115619 5018 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:26.094367 5035 consensus_queue.cc:260] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:26.116112 4922 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:26.218159 1568 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:26.207989 1490 leader_election.cc:334] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.952s (SENT)
I20260808 01:11:26.224197 1489 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.224375 1490 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:26.225189 5039 log.cc:924] Time spent T 00000000000000000000000000000000 P 5a043bcc76cc4126b15e712346791f23: Append to log took a long time: real 0.139s user 0.000s sys 0.002s
I20260808 01:11:26.216217 5038 raft_consensus.cc:2781] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:26.225975 5038 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:26.226287 5029 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:26.208698 5013 raft_consensus.cc:686] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:26.184881 4596 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:26.138635 5047 raft_consensus.cc:2781] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:11:26.278445 5047 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:26.278932 5047 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:11:26.279001 5013 consensus_queue.cc:237] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:26.220127 5018 raft_consensus.cc:3037] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:26.226524 5038 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:11:26.285949 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 78016.7us
I20260808 01:11:26.173951 4757 consensus_queue.cc:1046] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.109s
I20260808 01:11:26.226785 4731 raft_consensus.cc:2726] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260808 01:11:26.300698 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 59914.6us
I20260808 01:11:26.303489 5000 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:26.306249 5000 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:26.309546 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 103328us
I20260808 01:11:26.240461 4568 raft_consensus.cc:2726] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:26.286154 4935 raft_consensus.cc:2726] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:26.318588 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:26.322356 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 143554us
I20260808 01:11:26.324303 1489 leader_election.cc:398] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 7. Current term is 8.
I20260808 01:11:26.326292 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:26.300117 5032 raft_consensus.cc:2726] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:26.327874 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:26.328707 2208 raft_consensus.cc:2379] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
W20260808 01:11:26.329895 5050 log.cc:924] Time spent T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
I20260808 01:11:26.330677 4981 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:26.325662 5035 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:26.225222 5042 consensus_queue.cc:237] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 6.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:26.332827 2202 raft_consensus.cc:3032] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:26.333197 2202 raft_consensus.cc:729] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:26.333968 2202 consensus_queue.cc:260] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:26.334447 4553 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:26.335503 2204 raft_consensus.cc:1226] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:26.336560 5048 raft_consensus.cc:2726] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:26.286841 4936 consensus_queue.cc:1046] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260808 01:11:26.337970 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:26.336493 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:26.338897 1410 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:26.338200 1524 raft_consensus.cc:1226] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:26.341753 2200 raft_consensus.cc:2396] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260808 01:11:26.347626 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:26.350839 1328 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet a83a963773094d6db6495b7009fd8690 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6b09dbefd24b4d52b385c4dff5697dc5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:26.347127 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:26.367705 2202 raft_consensus.cc:3037] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:26.345732 4269 raft_consensus.cc:3032] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:11:26.381515 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:26.362052 4757 raft_consensus.cc:2781] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader election won for term 9
W20260808 01:11:26.398164 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 132963us
I20260808 01:11:26.367017 5060 raft_consensus.cc:2726] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:26.377153 4269 raft_consensus.cc:729] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
W20260808 01:11:26.403271 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 163286us
W20260808 01:11:26.403654 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 93321us
I20260808 01:11:26.408033 1489 leader_election.cc:398] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 8. Current term is 9.
I20260808 01:11:26.408317 1489 leader_election.cc:401] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:11:26.413522 2209 raft_consensus.cc:1226] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 9. Current term is 10. Ops: []
I20260808 01:11:26.413297 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:26.414402 1520 raft_consensus.cc:2379] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 5: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:26.414783 1490 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:26.417958 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:11:26.422819 4295 consensus_meta.cc:287] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.524s user 0.003s sys 0.007s
I20260808 01:11:26.424099 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:26.395815 4833 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:26.424821 5048 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:26.428036 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:26.473031 1519 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:26.432127 4891 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:26.432708 4922 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:26.438781 5018 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:26.440989 4757 raft_consensus.cc:686] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:26.442489 5060 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:26.444053 5029 consensus_queue.cc:1046] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:11:26.445087 5044 raft_consensus.cc:2726] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:26.447340 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:26.451932 1490 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.452524 4935 raft_consensus.cc:2726] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:26.453011 5049 consensus_queue.cc:1035] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20260808 01:11:26.453368 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 45476.1us
I20260808 01:11:26.455503 1522 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:26.519228 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:26.476336 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:26.481693 2209 raft_consensus.cc:1226] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:26.481316 5043 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:26.479558 4295 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:26.515558 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:26.501060 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:26.540107 1597 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 9.
I20260808 01:11:26.540114 1602 raft_consensus.cc:2379] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 7: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
W20260808 01:11:26.516705 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 98939.6us
I20260808 01:11:26.520352 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:26.459198 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:26.521446 1598 raft_consensus.cc:2379] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 7: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
W20260808 01:11:26.522202 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 289609us
I20260808 01:11:26.546780 1599 raft_consensus.cc:2379] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 5: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:26.501724 1603 raft_consensus.cc:1261] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260808 01:11:26.515024 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 63205.7us
I20260808 01:11:26.573783 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:26.577099 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:26.579792 1438 raft_consensus.cc:1226] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:26.584182 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:26.586086 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:26.586177 5072 consensus_queue.cc:1046] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:11:26.586807 1594 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 9.
I20260808 01:11:26.582065 4891 consensus_queue.cc:1046] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260808 01:11:26.587536 1440 raft_consensus.cc:2932] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 3 to 6, 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:26.588868 1443 raft_consensus.cc:1226] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:26.586287 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:11:26.582775 1410 connection.cc:232] RPC call timeout handler was delayed by -1.7385555510118138s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.38130000000000003s for a response
W20260808 01:11:26.590641 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 73969.6us
I20260808 01:11:26.589905 1447 raft_consensus.cc:2396] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:11:26.599793 2203 raft_consensus.cc:2354] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6.
I20260808 01:11:26.601704 1439 raft_consensus.cc:2379] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:26.604166 1438 raft_consensus.cc:1226] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260808 01:11:26.604754 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 85217.4us
I20260808 01:11:26.605751 1489 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:26.608814 1410 connection.cc:232] RPC call timeout handler was delayed by -1.1637609150202479s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.171s for a response
W20260808 01:11:26.609357 1410 connection.cc:232] RPC call timeout handler was delayed by -1.0152609150100034s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1875s for a response
I20260808 01:11:26.609405 4293 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:26.610201 1410 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:26.610272 4293 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:26.612223 5020 raft_consensus.cc:2781] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:26.612740 5020 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:26.613149 1410 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.613178 5020 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:26.616528 1410 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:26.617571 5054 raft_consensus.cc:2781] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:26.618045 5054 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:26.618512 5054 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:26.620326 4293 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:26.621872 5053 consensus_queue.cc:1046] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:11:26.419400 4269 consensus_queue.cc:260] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:26.625703 4269 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:26.625782 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:26.517207 4553 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:26.628185 4553 raft_consensus.cc:2726] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:26.628269 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:26.629853 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:26.490897 5030 raft_consensus.cc:2726] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:26.636683 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:26.638741 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:26.639402 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:26.642560 1522 raft_consensus.cc:2379] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 4: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:26.643548 1567 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:26.644969 1594 raft_consensus.cc:2379] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 7: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
W20260808 01:11:26.645258 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 62047.4us
I20260808 01:11:26.646152 2171 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.648581 2171 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.648952 1519 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:26.652721 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:26.653776 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:26.659389 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:26.661481 2206 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:26.651219 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:26.663187 1516 raft_consensus.cc:2454] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
W20260808 01:11:26.666790 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 50349us
I20260808 01:11:26.669054 1410 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:26.650354 2171 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:26.655428 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:26.678934 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:26.487819 4733 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:26.686231 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:26.683766 2171 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:26.691772 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:26.693532 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:26.696811 1522 raft_consensus.cc:3037] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:26.640594 4960 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:26.704330 1567 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:26.703348 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:26.588919 5032 consensus_queue.cc:1035] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.099s
I20260808 01:11:26.707320 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:26.674706 5044 raft_consensus.cc:2726] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:26.674441 4945 raft_consensus.cc:2726] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 01:11:26.680857 5015 consensus_meta.cc:287] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.567s user 0.010s sys 0.000s
I20260808 01:11:26.502014 5047 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:26.490725 5038 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:26.520052 5060 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:26.680233 4953 raft_consensus.cc:2726] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:26.726186 1410 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:26.755205 1410 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.691329 4886 raft_consensus.cc:2781] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:26.687927 5030 raft_consensus.cc:3037] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:26.729727 1522 raft_consensus.cc:2454] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
W20260808 01:11:26.751757 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 146119us
I20260808 01:11:26.682384 5065 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:26.760775 1444 raft_consensus.cc:1226] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
W20260808 01:11:26.758337 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 74675.6us
I20260808 01:11:26.765939 2171 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:26.760998 1489 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.772688 1596 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:11:26.757912 5042 raft_consensus.cc:2726] T 821d5636d01e4067bac2bbd7475921ef P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:26.773274 5085 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:26.773434 1596 raft_consensus.cc:595] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Received request to transfer leadership to TS 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:26.773834 5085 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:26.753139 4961 raft_consensus.cc:2781] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:26.768203 5053 raft_consensus.cc:2726] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:11:26.776904 1565 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:26.776901 5043 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:26.778826 5085 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:26.779146 5047 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:26.779815 5085 raft_consensus.cc:2726] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:26.781270 4295 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:26.756246 5015 raft_consensus.cc:2781] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:11:26.788241 5015 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:26.788795 5015 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:11:26.768726 2205 raft_consensus.cc:2454] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:26.752899 4757 consensus_queue.cc:237] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:11:26.798686 5058 log.cc:924] Time spent T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260808 01:11:26.799381 5065 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:26.805902 1448 raft_consensus.cc:1226] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
W20260808 01:11:26.822949 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 60237.7us
W20260808 01:11:26.834224 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 57410.8us
W20260808 01:11:26.840111 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 135849us
I20260808 01:11:26.815016 5073 raft_consensus.cc:2781] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:26.845922 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:26.714838 4960 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:26.847956 2171 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.853322 2171 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.853554 2208 raft_consensus.cc:1226] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260808 01:11:26.854463 1565 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:26.855052 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 94159.6us
I20260808 01:11:26.858018 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:26.832342 5038 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:26.859043 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 91343.8us
I20260808 01:11:26.865268 2206 raft_consensus.cc:1226] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 11 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20260808 01:11:26.865909 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:26.868002 1410 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:26.868304 2201 raft_consensus.cc:2362] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 7. Re-sending same reply.
I20260808 01:11:26.833767 4961 raft_consensus.cc:686] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:26.832947 4593 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:26.874444 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:26.871981 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:26.878347 2208 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:26.843223 4886 raft_consensus.cc:686] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:26.822857 5029 consensus_queue.cc:1035] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.034s
I20260808 01:11:26.874270 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:26.886282 1438 raft_consensus.cc:2379] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 7: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:26.875492 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:26.847270 5073 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:26.884969 2207 raft_consensus.cc:1261] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260808 01:11:26.890125 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:26.888310 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:26.890988 1448 raft_consensus.cc:2379] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:26.892303 1489 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.895915 1489 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:26.884132 5047 raft_consensus.cc:2781] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:11:26.905596 1489 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:26.908531 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:26.917124 1438 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:26.910137 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:26.888110 4593 raft_consensus.cc:2726] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 01:11:26.915817 2202 consensus_meta.cc:287] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.547s user 0.006s sys 0.005s
I20260808 01:11:26.920912 1442 raft_consensus.cc:2379] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 4: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
W20260808 01:11:26.914161 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 206705us
I20260808 01:11:26.908805 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:26.922612 1447 raft_consensus.cc:2379] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 7: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:26.922506 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:26.920049 1439 raft_consensus.cc:2454] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:26.923455 1444 raft_consensus.cc:2379] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:26.921459 2202 raft_consensus.cc:1261] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:11:26.925799 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:26.925904 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:26.894661 5073 raft_consensus.cc:3037] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:26.895632 5042 raft_consensus.cc:2726] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:11:26.919194 5031 consensus_queue.cc:1046] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.093s
I20260808 01:11:26.916141 5047 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:26.910809 5072 raft_consensus.cc:2726] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:26.926921 5089 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:26.927007 5026 raft_consensus.cc:2726] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:11:26.927062 1489 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:26.976049 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 49125.9us
I20260808 01:11:26.976395 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:26.929193 4960 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:26.930613 5054 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:26.883626 5038 consensus_queue.cc:1046] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260808 01:11:26.931277 5088 raft_consensus.cc:2781] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:26.931722 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:26.988329 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:26.969638 5047 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:11:26.932053 5034 consensus_queue.cc:1046] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260808 01:11:26.932406 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:26.940469 5065 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:26.940832 5082 log.cc:924] Time spent T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260808 01:11:26.869340 5070 consensus_queue.cc:1035] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.073s
I20260808 01:11:26.941980 4886 consensus_queue.cc:237] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:26.897066 5060 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:26.934151 1448 raft_consensus.cc:2379] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 10: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:26.951502 1443 raft_consensus.cc:2379] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 6: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:26.953140 1490 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:26.953672 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:26.954030 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:26.928722 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:27.008456 1438 raft_consensus.cc:2379] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:26.926012 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:26.967499 1442 raft_consensus.cc:1226] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 11 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 10. Current term is 11. Ops: []
I20260808 01:11:26.956223 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:26.981943 5087 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:26.981719 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 127341us
I20260808 01:11:27.013211 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:26.983075 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:26.985493 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:27.017102 1522 raft_consensus.cc:3037] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:27.017210 1516 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:27.018240 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:11:26.975538 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 127065us
I20260808 01:11:27.019704 1568 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:26.989480 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:26.931788 5086 raft_consensus.cc:2726] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:27.023222 4731 raft_consensus.cc:3032] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:27.023787 4731 raft_consensus.cc:729] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:26.933173 5068 raft_consensus.cc:2781] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:27.029140 5068 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:27.029661 5068 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:26.999302 1439 raft_consensus.cc:2379] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 4: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:26.948630 4757 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:27.033195 4757 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:26.948311 4961 consensus_queue.cc:237] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:27.011026 5048 consensus_meta.cc:287] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.515s user 0.013s sys 0.000s
I20260808 01:11:27.040854 5048 raft_consensus.cc:2726] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675) with higher term. Message: Invalid argument: T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 8. Current term is 9.
I20260808 01:11:27.041853 1525 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 9.
I20260808 01:11:26.980533 5089 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:27.004136 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 145239us
I20260808 01:11:27.047149 5054 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:26.933347 5092 raft_consensus.cc:2726] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 01:11:26.973013 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 119787us
I20260808 01:11:27.048509 2171 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:26.933162 5020 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:26.994400 5090 raft_consensus.cc:2726] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 01:11:26.994133 4664 consensus_meta.cc:287] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 1.099s user 0.013s sys 0.004s
I20260808 01:11:27.050042 4935 raft_consensus.cc:2726] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:27.050745 5020 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:27.051486 1446 raft_consensus.cc:2379] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:27.052183 2209 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 7.
W20260808 01:11:27.013142 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 60083.2us
I20260808 01:11:27.054618 1410 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:27.057351 2171 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:27.058401 1490 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:27.058615 1410 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:27.065469 1447 raft_consensus.cc:1226] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260808 01:11:27.067948 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:27.068655 2172 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:27.068737 1517 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 8.
I20260808 01:11:27.068837 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:27.069855 2202 raft_consensus.cc:2396] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260808 01:11:27.070698 1522 raft_consensus.cc:2454] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:27.003302 5088 raft_consensus.cc:686] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
W20260808 01:11:27.015784 5035 consensus_meta.cc:287] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.684s user 0.011s sys 0.002s
I20260808 01:11:27.072183 2171 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:27.072849 1490 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:27.074657 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:27.074590 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:27.075982 1568 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:27.074779 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:27.078470 2171 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:27.079424 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:27.080240 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:27.083626 5100 raft_consensus.cc:2726] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:11:27.083854 5090 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:27.084266 5020 raft_consensus.cc:2726] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:27.085009 5083 raft_consensus.cc:2726] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:27.085793 5074 raft_consensus.cc:2781] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:27.086422 5048 raft_consensus.cc:2726] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:11:27.086596 5065 raft_consensus.cc:2781] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:27.087054 5065 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:27.087424 5065 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:27.087164 5048 consensus_queue.cc:1046] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:11:27.048583 5089 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:27.088198 5048 consensus_queue.cc:1046] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260808 01:11:27.088519 5089 raft_consensus.cc:3032] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:11:27.085716 1487 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:27.089226 1519 raft_consensus.cc:2396] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260808 01:11:27.090322 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:27.091528 2171 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:27.093113 2206 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260808 01:11:27.095101 1490 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:27.069693 5070 raft_consensus.cc:2781] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:27.070278 4664 raft_consensus.cc:2726] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:27.099642 1410 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:27.092734 5089 raft_consensus.cc:729] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:27.102519 1490 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:27.103099 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:27.103667 1490 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:27.071655 4757 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:27.024969 4731 consensus_queue.cc:260] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:27.078915 5060 raft_consensus.cc:2726] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:11:27.109997 2171 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:27.111117 1410 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:27.119477 5044 raft_consensus.cc:2781] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:27.121399 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:27.121513 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:27.121737 5031 consensus_queue.cc:1046] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260808 01:11:27.122913 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:27.123674 1443 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 8.
I20260808 01:11:27.124061 1442 raft_consensus.cc:1261] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:11:27.124454 5088 consensus_queue.cc:237] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:27.125238 1442 raft_consensus.cc:1226] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:27.128069 1523 raft_consensus.cc:1261] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:11:27.129233 4731 raft_consensus.cc:3037] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:27.129513 1520 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260808 01:11:27.131817 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:27.114725 5042 raft_consensus.cc:2726] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:11:27.103500 5092 raft_consensus.cc:2781] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:27.133665 5092 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:27.134007 5092 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:27.103327 5026 consensus_queue.cc:1035] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260808 01:11:27.136389 5034 raft_consensus.cc:2726] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:27.137158 5048 raft_consensus.cc:2781] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:27.124061 2206 raft_consensus.cc:1261] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:11:27.138715 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:27.139202 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:27.139277 5026 consensus_queue.cc:1035] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:27.090109 5072 consensus_queue.cc:1046] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20260808 01:11:27.140786 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:27.101686 5070 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:27.141530 5070 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:27.120344 5044 raft_consensus.cc:686] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:27.143891 4935 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:27.143177 5044 consensus_queue.cc:237] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:27.108604 5089 consensus_queue.cc:260] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:27.139277 1596 raft_consensus.cc:3037] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:27.122962 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:27.137570 5048 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:27.140841 5072 raft_consensus.cc:3032] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:11:27.151094 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:27.152659 1410 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:27.101548 5074 raft_consensus.cc:686] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:27.160894 1440 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 6 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260808 01:11:27.157092 2172 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:27.056595 5087 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:27.146566 5089 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:27.118721 5032 raft_consensus.cc:2726] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:27.163113 2172 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:27.114470 5020 raft_consensus.cc:2726] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:27.164989 1410 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:27.277283 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 114299us
I20260808 01:11:27.153692 5048 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:27.153963 5072 raft_consensus.cc:729] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:27.175191 5029 raft_consensus.cc:2726] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260808 01:11:27.302553 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 183030us
I20260808 01:11:27.173652 5060 raft_consensus.cc:2726] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 01:11:27.301221 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 199465us
I20260808 01:11:27.277251 5038 raft_consensus.cc:3032] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:27.315881 5038 raft_consensus.cc:729] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:27.317065 5038 consensus_queue.cc:260] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:27.318753 5038 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:11:27.399708 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 234802us
I20260808 01:11:27.401453 1410 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:27.160410 5015 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:27.304313 5085 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:11:27.405874 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 77284.9us
I20260808 01:11:27.427878 1565 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:27.400960 5029 raft_consensus.cc:2726] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260808 01:11:27.444402 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 133193us
I20260808 01:11:27.433424 5032 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:27.474128 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:27.424078 5042 consensus_queue.cc:1035] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
W20260808 01:11:27.454797 4269 consensus_meta.cc:287] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.829s user 0.013s sys 0.004s
I20260808 01:11:27.405225 5013 consensus_queue.cc:1046] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260808 01:11:27.431430 4664 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
I20260808 01:11:27.487273 1440 raft_consensus.cc:1226] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
W20260808 01:11:27.493991 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 93260.3us
I20260808 01:11:27.444410 4757 raft_consensus.cc:2781] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:27.494870 1410 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:27.501214 1603 raft_consensus.cc:2396] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260808 01:11:27.493394 1521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2178) took 1563 ms. Trace:
W20260808 01:11:27.489498 5102 log.cc:924] Time spent T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.182s user 0.001s sys 0.000s
I20260808 01:11:27.432006 5085 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:27.503270 1521 rpcz_store.cc:276] 0808 01:11:25.929628 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:25.929747 (+ 119us) service_pool.cc:227] Handling call
0808 01:11:25.945505 (+ 15758us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:25.980586 (+ 35081us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:25.980631 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:25.980685 (+ 54us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:25.980735 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:27.493105 (+1512370us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b540035cad4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:27.493166 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:27.493311 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":80099712}
I20260808 01:11:27.486500 5032 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:27.474149 5026 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:27.491456 5030 consensus_meta.cc:287] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.721s user 0.010s sys 0.002s
I20260808 01:11:27.508625 1410 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:27.516676 1524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2188) took 1387 ms. Trace:
I20260808 01:11:27.517565 1524 rpcz_store.cc:276] 0808 01:11:26.129182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:26.199313 (+ 70131us) service_pool.cc:227] Handling call
0808 01:11:26.222190 (+ 22877us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:26.343444 (+121254us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:27.516449 (+1173005us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b540034bf54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:27.516616 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":266339200}
I20260808 01:11:27.489521 5097 raft_consensus.cc:2781] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260808 01:11:27.523186 1444 consensus_meta.cc:287] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.511s user 0.009s sys 0.000s
I20260808 01:11:27.524461 5110 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.081s
I20260808 01:11:27.507548 5116 consensus_queue.cc:1046] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260808 01:11:27.525774 5108 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:27.529469 5113 log.cc:924] Time spent T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
W20260808 01:11:27.530381 5109 log.cc:924] Time spent T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20260808 01:11:27.490497 5062 raft_consensus.cc:2726] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:27.533413 1410 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:27.492272 5091 raft_consensus.cc:2726] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:11:27.541733 4731 raft_consensus.cc:2726] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:27.541873 5087 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:27.547490 5029 raft_consensus.cc:2726] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:27.534849 1596 raft_consensus.cc:2454] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:27.548058 5090 raft_consensus.cc:2726] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:27.479323 5083 consensus_queue.cc:1046] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260808 01:11:27.436403 5072 consensus_queue.cc:260] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:27.549474 5000 raft_consensus.cc:3032] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Stepping down as leader of term 3
W20260808 01:11:27.550697 5068 consensus_meta.cc:287] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.520s user 0.009s sys 0.001s
W20260808 01:11:27.547017 5073 consensus_meta.cc:287] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.576s user 0.008s sys 0.000s
I20260808 01:11:27.512487 5020 raft_consensus.cc:2726] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:11:27.552857 5026 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:27.515758 5111 consensus_queue.cc:1035] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.128s
I20260808 01:11:27.511806 4757 raft_consensus.cc:686] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:27.540925 4871 consensus_queue.cc:1035] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:27.542975 5085 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:27.535055 1444 raft_consensus.cc:2454] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:11:27.542374 4568 raft_consensus.cc:3032] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:27.578533 2200 raft_consensus.cc:1226] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 3. Current term is 4. Ops: []
I20260808 01:11:27.651783 5000 raft_consensus.cc:729] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:27.535557 5030 raft_consensus.cc:2726] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260808 01:11:27.584096 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 50783.5us
I20260808 01:11:27.691287 2208 raft_consensus.cc:1261] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
W20260808 01:11:27.590804 1490 leader_election.cc:334] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.818s (SENT)
I20260808 01:11:27.692065 1490 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:27.691676 2208 pending_rounds.cc:76] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd: Aborting all ops after (but not including) 5
I20260808 01:11:27.693127 2208 pending_rounds.cc:97] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 6.6
W20260808 01:11:27.692911 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 102172us
I20260808 01:11:27.693472 2208 raft_consensus.cc:2894] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Aborting config change with OpId 6.6: Aborted: Op aborted by new leader
I20260808 01:11:27.693948 2208 raft_consensus.cc:2866] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [term 9 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:11:27.694021 1490 leader_election.cc:334] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291): Timed out: RequestConsensusVote RPC to 127.31.212.65:38291 timed out after 1.983s (SENT)
I20260808 01:11:27.694290 2208 pending_rounds.cc:97] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20260808 01:11:27.694597 2208 raft_consensus.cc:2866] T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd [term 9 LEARNER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:11:27.643783 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 494433us
I20260808 01:11:27.646580 1448 raft_consensus.cc:1226] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 9. Current term is 10. Ops: []
I20260808 01:11:27.584069 5095 raft_consensus.cc:2781] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:27.651434 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:27.695667 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:27.696210 5095 raft_consensus.cc:686] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:27.696280 1447 raft_consensus.cc:2454] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
W20260808 01:11:27.647284 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 219513us
I20260808 01:11:27.697214 1565 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:27.440372 5074 consensus_queue.cc:237] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:27.698251 1518 raft_consensus.cc:1261] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:11:27.585270 5092 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:27.683943 2201 raft_consensus.cc:1226] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 11 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20260808 01:11:27.682832 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:27.702491 1524 raft_consensus.cc:2454] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:27.650497 5015 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:27.534511 5097 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:11:27.663758 5086 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.510s user 0.006s sys 0.012s
I20260808 01:11:27.691217 5000 consensus_queue.cc:260] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:27.694546 1490 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:27.697701 1523 raft_consensus.cc:1226] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:27.721792 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:27.654513 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:27.699272 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:11:27.644430 4961 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.010s sys 0.002s
W20260808 01:11:27.725036 4961 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s user 0.010s sys 0.002s
I20260808 01:11:27.687625 4568 raft_consensus.cc:729] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:27.704737 1410 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:27.756330 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 51670.3us
I20260808 01:11:27.757144 1410 leader_election.cc:302] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:27.707368 1524 raft_consensus.cc:1261] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 01:11:27.687191 5072 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:11:27.582332 5108 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:27.765108 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:27.704597 2171 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:27.692886 5085 raft_consensus.cc:2726] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:27.765298 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:27.766201 1447 raft_consensus.cc:2379] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 8: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:27.725067 5000 raft_consensus.cc:3037] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:11:27.718750 4269 raft_consensus.cc:3032] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Stepping down as leader of term 4
W20260808 01:11:27.766623 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 62161us
W20260808 01:11:27.721649 5023 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260808 01:11:27.719800 5097 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:27.779940 1524 raft_consensus.cc:1261] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:11:27.783720 1490 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:27.782028 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:27.771301 5047 consensus_meta.cc:287] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.764s user 0.010s sys 0.002s
I20260808 01:11:27.660221 5068 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:27.660771 5073 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:27.788450 1487 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:27.788223 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 92860.2us
I20260808 01:11:27.756026 4731 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:27.767289 5029 raft_consensus.cc:2726] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:11:27.769649 5015 raft_consensus.cc:2726] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:11:27.773217 5087 raft_consensus.cc:2781] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:27.771209 4871 raft_consensus.cc:2726] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260808 01:11:27.780495 4935 consensus_meta.cc:287] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.636s user 0.011s sys 0.001s
W20260808 01:11:27.780882 5070 consensus_meta.cc:287] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.639s user 0.012s sys 0.000s
I20260808 01:11:27.905906 5092 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:27.902320 4891 raft_consensus.cc:3032] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:11:27.833019 5026 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:27.904496 5073 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:27.903718 1445 raft_consensus.cc:3032] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:27.985345 1445 raft_consensus.cc:729] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:27.907222 5108 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:27.799034 4757 consensus_queue.cc:237] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:27.792589 5032 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:27.992169 5032 raft_consensus.cc:482] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:27.992763 5032 raft_consensus.cc:504] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:27.995081 5032 leader_election.cc:288] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:27.903129 5068 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:27.799609 4295 consensus_queue.cc:1035] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260808 01:11:27.834307 4731 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:27.996313 5119 consensus_queue.cc:1035] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:11:27.997370 5068 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:27.997720 4731 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:27.897742 5034 raft_consensus.cc:979] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78: : Instructing follower 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:11:27.997920 5068 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:27.998755 5034 raft_consensus.cc:1067] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Signalling peer 95b661b03f1d460b9657e61d4b0247cd to start an election
I20260808 01:11:27.827049 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:27.999886 2201 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:28.000164 5026 raft_consensus.cc:2726] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:27.905555 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:11:27.925109 5089 consensus_meta.cc:287] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.739s user 0.012s sys 0.000s
I20260808 01:11:27.916383 5037 raft_consensus.cc:3032] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:28.001098 5037 raft_consensus.cc:729] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:28.002028 5037 consensus_queue.cc:260] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.003305 5037 raft_consensus.cc:3037] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:28.004261 1520 raft_consensus.cc:2396] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260808 01:11:27.799767 5023 consensus_queue.cc:1035] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.090s
W20260808 01:11:27.873853 5048 consensus_meta.cc:287] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.566s user 0.010s sys 0.002s
I20260808 01:11:28.008438 5048 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.009887 5068 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:27.795717 4886 consensus_queue.cc:1046] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.092s
I20260808 01:11:27.922243 4733 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:27.793916 5053 consensus_queue.cc:1035] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.092s
I20260808 01:11:27.927752 1516 raft_consensus.cc:1226] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 4. Current term is 5. Ops: []
W20260808 01:11:27.899039 5130 log.cc:924] Time spent T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.173s user 0.001s sys 0.000s
I20260808 01:11:27.878970 5083 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:27.827967 4568 consensus_queue.cc:260] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:27.922590 5065 consensus_meta.cc:287] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.835s user 0.006s sys 0.003s
I20260808 01:11:27.800697 5047 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:27.832818 5103 raft_consensus.cc:3032] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:11:27.825469 4953 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:28.045171 4953 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:28.045215 5065 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.047330 4953 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:27.775036 5138 raft_consensus.cc:2726] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:11:27.874704 1597 raft_consensus.cc:1226] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: []
W20260808 01:11:27.876554 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 74170us
I20260808 01:11:27.910043 5127 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:27.778103 4269 raft_consensus.cc:729] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:28.032951 5083 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:27.825013 5123 log.cc:924] Time spent T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260808 01:11:27.909212 2209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1165) took 1063 ms. Trace:
I20260808 01:11:27.755270 5044 consensus_queue.cc:1046] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260808 01:11:27.723109 5088 consensus_queue.cc:1046] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.070s
I20260808 01:11:27.697216 5095 consensus_queue.cc:237] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.044950 5103 raft_consensus.cc:729] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:28.034906 5048 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:28.050143 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:11:27.965376 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 268203us
W20260808 01:11:27.966061 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 177702us
W20260808 01:11:27.970526 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 186858us
I20260808 01:11:27.974684 5087 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:28.054432 2209 rpcz_store.cc:276] 0808 01:11:26.845435 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:26.845600 (+ 165us) service_pool.cc:227] Handling call
0808 01:11:27.908989 (+1063389us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b540067ded4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:27.909117 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":470610304}
I20260808 01:11:28.073040 1448 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260808 01:11:28.029253 5092 raft_consensus.cc:2726] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:27.981603 5070 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:28.014153 1525 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58266 (request call id 2270) took 1601 ms (client timeout 1660 ms). Trace:
I20260808 01:11:28.077136 1490 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:28.015189 2203 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 783) took 1689 ms (client timeout 1614 ms). Trace:
I20260808 01:11:27.982048 4935 raft_consensus.cc:2726] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:28.035916 1602 raft_consensus.cc:1261] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:11:28.040820 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:28.086745 2200 raft_consensus.cc:1261] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:11:28.041528 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:28.081666 2203 rpcz_store.cc:269] 0808 01:11:26.325906 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:26.326045 (+ 139us) service_pool.cc:227] Handling call
0808 01:11:28.014917 (+1688872us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b540004c754. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:28.015100 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":627102976}
I20260808 01:11:28.040820 1594 raft_consensus.cc:1261] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260808 01:11:28.043419 5018 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:28.044163 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:28.088433 1604 raft_consensus.cc:2379] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:28.060662 1410 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:28.049288 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:28.049504 2202 raft_consensus.cc:1226] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:28.049808 1440 raft_consensus.cc:1226] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:27.980831 4891 raft_consensus.cc:729] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:28.073942 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:28.096710 1444 raft_consensus.cc:2379] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 7: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:28.070538 1524 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 7.
I20260808 01:11:28.070833 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:28.097824 1599 raft_consensus.cc:2379] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 4: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
W20260808 01:11:28.081086 1525 rpcz_store.cc:269] 0808 01:11:26.412971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:26.413274 (+ 303us) service_pool.cc:227] Handling call
0808 01:11:28.013868 (+1600594us) spinlock_profiling.cc:243] Waited 572 ms on lock 0x7b540012eb54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:28.014033 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1315628416}
I20260808 01:11:28.085327 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:28.085541 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:28.099323 2206 raft_consensus.cc:2379] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 6: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:28.092531 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:28.100121 2203 raft_consensus.cc:2379] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 8: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:28.087532 1598 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:28.055161 5044 raft_consensus.cc:3032] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:11:28.120236 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:28.051002 5047 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:28.122231 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 45189.1us
I20260808 01:11:28.135789 1490 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:28.075387 5087 raft_consensus.cc:3037] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:28.047802 5065 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:28.120380 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:28.081310 5049 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:28.139142 2209 raft_consensus.cc:1226] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:28.142982 1568 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:28.143260 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:28.143937 1602 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 8.
I20260808 01:11:28.144440 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:28.144747 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:28.145241 2209 raft_consensus.cc:2454] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:28.141736 1522 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:11:28.123931 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:28.148655 1599 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:11:28.149735 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:28.153164 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:28.155948 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:28.156770 1598 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:28.156966 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:28.157786 1604 raft_consensus.cc:2379] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 7: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:28.158013 1490 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:28.158875 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:28.131322 2171 leader_election.cc:302] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:28.054510 5083 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:28.122372 1487 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:28.134215 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:28.051970 5125 consensus_queue.cc:1046] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.082s
I20260808 01:11:28.119524 4886 raft_consensus.cc:2781] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:28.119524 5053 raft_consensus.cc:482] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:28.133426 4568 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:28.163800 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:28.170787 1516 raft_consensus.cc:1226] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:28.125468 5044 raft_consensus.cc:729] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:28.059540 5103 consensus_queue.cc:260] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:28.053359 4269 consensus_queue.cc:260] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:28.178087 5048 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:28.178541 5070 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:28.178664 5032 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:28.232556 5032 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:28.233652 1603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2384) took 1660 ms (client timeout 1805 ms). Trace:
W20260808 01:11:28.234068 1603 rpcz_store.cc:269] 0808 01:11:26.573318 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:26.573459 (+ 141us) service_pool.cc:227] Handling call
0808 01:11:28.233376 (+1659917us) spinlock_profiling.cc:243] Waited 926 ms on lock 0x7b540028f5d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:28.233563 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2130386176}
I20260808 01:11:28.234948 5032 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:28.180184 4886 raft_consensus.cc:686] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:28.235280 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:28.236490 1520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58280 (request call id 2203) took 1105 ms. Trace:
I20260808 01:11:28.159758 5049 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.181094 2206 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:28.236879 1520 rpcz_store.cc:276] 0808 01:11:27.131396 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:27.131557 (+ 161us) service_pool.cc:227] Handling call
0808 01:11:28.236293 (+1104736us) spinlock_profiling.cc:243] Waited 871 ms on lock 0x7b54001a2054. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:28.236433 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2004003200}
I20260808 01:11:28.237998 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:28.180637 5137 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:28.175096 5138 raft_consensus.cc:2726] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:28.052215 5127 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.181206 5135 raft_consensus.cc:2726] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:11:28.138470 5034 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
W20260808 01:11:28.182305 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 60054.3us
I20260808 01:11:28.057312 5097 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.183132 5141 raft_consensus.cc:2781] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:28.193089 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:28.193840 5124 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:28.197017 1521 raft_consensus.cc:1261] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:11:28.252839 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:28.198179 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:28.253628 1603 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:11:28.199757 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:28.201287 5053 raft_consensus.cc:504] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:28.203475 1598 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:28.085428 5088 consensus_queue.cc:1046] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.074s
I20260808 01:11:28.206372 1522 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 10.
I20260808 01:11:28.056161 5140 consensus_queue.cc:1046] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.106s
I20260808 01:11:28.253479 1521 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:28.213450 1565 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:28.173928 2209 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563"
dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:49260
I20260808 01:11:28.209504 1599 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 10.
I20260808 01:11:28.223869 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:28.149705 5094 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:28.227631 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:28.229820 5044 consensus_queue.cc:260] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.236074 4886 consensus_queue.cc:237] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:28.193089 5015 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:28.138119 5143 raft_consensus.cc:2781] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:11:28.250880 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:28.142530 5040 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:28.206552 5065 raft_consensus.cc:2781] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:28.263304 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:28.264015 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:28.238744 1520 raft_consensus.cc:2379] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:28.283766 2209 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:11:28.249042 5141 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:28.309139 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:28.313917 1598 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:28.327899 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccc57605ae24bf4bacee4e4741bbe95" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:28.328670 1599 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:28.238356 5049 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:28.310452 1600 raft_consensus.cc:2362] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6. Re-sending same reply.
I20260808 01:11:28.264657 5032 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:28.300884 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 87529.4us
I20260808 01:11:28.259738 4731 consensus_queue.cc:1046] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
I20260808 01:11:28.281735 5124 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.183229 5145 raft_consensus.cc:482] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:28.297956 5108 consensus_queue.cc:1046] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.093s
I20260808 01:11:28.239018 5137 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.285881 5125 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.096s
I20260808 01:11:28.295466 4891 consensus_queue.cc:260] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:28.257028 5138 consensus_queue.cc:1035] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.130s
W20260808 01:11:28.311373 5000 consensus_meta.cc:287] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.521s user 0.011s sys 0.000s
I20260808 01:11:28.242164 5127 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:28.315555 2209 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:28.318863 2206 raft_consensus.cc:1261] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260808 01:11:28.161159 5147 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:28.325404 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:28.377583 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:28.349951 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:28.360693 2202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 1050) took 1289 ms. Trace:
W20260808 01:11:28.366199 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 642516us
I20260808 01:11:28.338891 2171 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:28.339340 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:28.597361 1440 raft_consensus.cc:1226] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:28.376683 1444 raft_consensus.cc:1226] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260808 01:11:28.611904 1594 raft_consensus.cc:3037] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:28.354533 1519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58280 (request call id 2193) took 1782 ms. Trace:
I20260808 01:11:28.315158 1603 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 10.
I20260808 01:11:28.367723 1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2393) took 1229 ms. Trace:
I20260808 01:11:28.612685 1519 rpcz_store.cc:276] 0808 01:11:26.572096 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:26.652452 (+ 80356us) service_pool.cc:227] Handling call
0808 01:11:28.354264 (+1701812us) spinlock_profiling.cc:243] Waited 435 ms on lock 0x7b540023c3d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:28.354456 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1001624320}
I20260808 01:11:28.364056 2205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 1049) took 1739 ms. Trace:
I20260808 01:11:28.294961 5111 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:28.578742 5129 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
W20260808 01:11:28.459306 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 124001us
I20260808 01:11:28.352579 5044 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:11:28.619778 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:28.338181 5141 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:28.371289 1599 raft_consensus.cc:2454] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:28.339720 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:28.379212 5065 raft_consensus.cc:686] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [term 7 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:28.676424 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:28.677924 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:28.678609 1447 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:28.326414 5015 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:28.683813 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:28.643854 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:28.650107 5037 raft_consensus.cc:2781] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:11:28.685164 2206 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 7.
I20260808 01:11:28.613699 2202 rpcz_store.cc:276] 0808 01:11:27.070943 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:27.081646 (+ 10703us) service_pool.cc:227] Handling call
0808 01:11:27.101687 (+ 20041us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:28.093930 (+992243us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:28.360385 (+266455us) spinlock_profiling.cc:243] Waited 947 ms on lock 0x7b540060d954. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:28.360626 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2179233152}
I20260808 01:11:28.614141 2205 rpcz_store.cc:276] 0808 01:11:26.624373 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:26.624562 (+ 189us) service_pool.cc:227] Handling call
0808 01:11:28.363867 (+1739305us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b5400708fd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:28.363984 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":326899072}
I20260808 01:11:28.388422 1445 consensus_queue.cc:260] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:28.686543 1445 raft_consensus.cc:3037] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:28.620888 1565 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:28.313449 4553 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:28.648487 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:28.311172 5094 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.651638 1567 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:28.651773 1444 raft_consensus.cc:1261] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:11:28.606561 1598 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:28.352952 5112 raft_consensus.cc:3032] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:11:28.362845 4960 raft_consensus.cc:3032] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:28.620577 2208 raft_consensus.cc:2454] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:28.658528 1440 raft_consensus.cc:1226] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:28.342104 5099 raft_consensus.cc:2781] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:28.594929 5053 leader_election.cc:288] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:28.376008 5089 raft_consensus.cc:3032] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Stepping down as leader of term 6
W20260808 01:11:28.616343 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 277541us
I20260808 01:11:28.337357 5040 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.682335 1439 raft_consensus.cc:1261] T 7a2611aecd26485e8912ab55890ec0fa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:11:28.616888 5032 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:28.613368 1597 rpcz_store.cc:276] 0808 01:11:27.138006 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:27.138120 (+ 114us) service_pool.cc:227] Handling call
0808 01:11:27.138465 (+ 345us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:28.048979 (+910514us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:28.367423 (+318444us) spinlock_profiling.cc:243] Waited 736 ms on lock 0x7b54002099d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:28.367648 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1692683520}
I20260808 01:11:28.313779 5097 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:28.689447 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:28.690145 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:11:28.690593 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 69815.6us
I20260808 01:11:28.693176 1408 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:28.695035 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:11:28.698319 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 46771.5us
I20260808 01:11:28.485056 5049 consensus_queue.cc:1035] T 2355fe5dfc824ec78c044695be9d32de P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.190s
I20260808 01:11:28.704021 2171 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:28.616318 5127 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:28.611347 1521 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 9.
I20260808 01:11:28.703398 5089 raft_consensus.cc:729] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
W20260808 01:11:28.726974 5152 log.cc:924] Time spent T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260808 01:11:28.727295 5094 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:28.744557 2171 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:28.700009 5099 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:11:28.729388 5154 log.cc:924] Time spent T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20260808 01:11:28.732268 1598 raft_consensus.cc:2379] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:28.751546 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:28.729986 5040 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:28.699846 4553 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.703624 5031 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:28.696203 5112 raft_consensus.cc:729] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:28.735847 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:28.614879 5147 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.740141 5000 raft_consensus.cc:2781] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:28.732694 1447 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 9.
I20260808 01:11:28.739049 1524 raft_consensus.cc:1226] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 8. Current term is 9. Ops: [6.7-8.8]
I20260808 01:11:28.688385 5037 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:28.614662 5145 raft_consensus.cc:504] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:28.654034 5111 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.665793 5129 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.743908 1567 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:28.750887 5099 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:11:28.728449 5153 log.cc:924] Time spent T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
W20260808 01:11:28.750303 2207 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.005s sys 0.026s
W20260808 01:11:28.753398 1410 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.014s sys 0.022s
I20260808 01:11:28.760721 1445 raft_consensus.cc:1261] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260808 01:11:28.741446 5087 consensus_meta.cc:287] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.577s user 0.009s sys 0.000s
W20260808 01:11:28.684417 5085 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.007s sys 0.013s
I20260808 01:11:28.696559 5108 raft_consensus.cc:2726] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:28.817445 1445 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55040 (request call id 2315) took 2234 ms. Trace:
I20260808 01:11:28.817849 1445 rpcz_store.cc:276] 0808 01:11:26.576027 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:26.576149 (+ 122us) service_pool.cc:227] Handling call
0808 01:11:26.576446 (+ 297us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:27.903628 (+1327182us) spinlock_profiling.cc:243] Waited 98.7 ms on lock 0x7b540033dea0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34c9cf730 00007fe34ca41300 00007fe34ca42371 00007fe34ca3eedf 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc
0808 01:11:28.383635 (+480007us) spinlock_profiling.cc:243] Waited 47.7 ms on lock 0x7b540033dea0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34c9d5951 00007fe34ca36321 00007fe34ca37b4e 00007fe34ca41c89 00007fe34ca42556 00007fe34ca3eedf 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612
0808 01:11:28.809735 (+426100us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:28.809944 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":336725248}
I20260808 01:11:28.689635 5135 consensus_queue.cc:1046] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
W20260808 01:11:28.778777 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 440149us
I20260808 01:11:28.775182 1600 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7"
dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
mode: GRACEFUL
new_leader_uuid: "95b661b03f1d460b9657e61d4b0247cd"
from {username='slave'} at 127.0.0.1:51424
I20260808 01:11:28.779675 1565 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:28.701586 4960 raft_consensus.cc:729] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
W20260808 01:11:28.802929 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 59129.9us
I20260808 01:11:28.804495 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:28.756877 5097 raft_consensus.cc:2726] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:11:28.809399 2207 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.653s user 0.005s sys 0.026s
W20260808 01:11:28.809456 1410 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.627s user 0.014s sys 0.022s
W20260808 01:11:28.828783 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 693894us
I20260808 01:11:28.787132 5037 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:11:28.831017 1565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 51477.5us
W20260808 01:11:28.831354 1410 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 1.805s (SENT) [suppressed 31 similar messages]
W20260808 01:11:28.834935 1604 consensus_meta.cc:287] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.623s user 0.009s sys 0.000s
I20260808 01:11:28.760799 5094 raft_consensus.cc:2781] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:28.826114 1567 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:28.826598 1489 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:28.861012 1489 leader_election.cc:302] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:28.827647 4922 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
W20260808 01:11:28.819206 5085 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s user 0.007s sys 0.013s
I20260808 01:11:28.835978 1597 raft_consensus.cc:1261] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260808 01:11:28.837081 5155 log.cc:924] Time spent T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.135s user 0.001s sys 0.000s
I20260808 01:11:28.837317 5129 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:28.748670 5127 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.838203 5111 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:28.838650 5032 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:28.839391 5015 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:28.762985 5138 consensus_queue.cc:1035] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260808 01:11:28.839975 5020 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:28.840498 5133 raft_consensus.cc:2726] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:28.936482 1597 raft_consensus.cc:1226] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 6. Current term is 7. Ops: [6.8-6.8]
W20260808 01:11:28.936691 5151 log.cc:924] Time spent T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.101s user 0.000s sys 0.001s
I20260808 01:11:28.937692 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:28.761627 5086 consensus_queue.cc:1035] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.063s
I20260808 01:11:28.890133 5098 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:28.940106 5098 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.942641 5165 raft_consensus.cc:2726] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:11:28.841130 5112 consensus_queue.cc:260] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:28.944854 5040 consensus_queue.cc:1035] T 921f3eb5150b4105a4b68355cca8cb7a P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:11:28.764251 5065 consensus_queue.cc:237] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.948467 5127 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:28.893811 5094 raft_consensus.cc:686] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:28.950217 5094 consensus_queue.cc:237] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.818996 5087 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.952033 5103 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:28.953471 5087 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:28.953877 5110 raft_consensus.cc:3032] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:28.954393 5110 raft_consensus.cc:729] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:28.747141 5062 consensus_queue.cc:1046] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.061s
I20260808 01:11:28.955808 5166 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:28.956197 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:28.956668 5062 raft_consensus.cc:3032] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:28.956990 2201 raft_consensus.cc:2379] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 6: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:28.843600 1594 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 8.
I20260808 01:11:28.957183 5062 raft_consensus.cc:729] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:28.957646 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:28.958093 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:28.854180 1410 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:28.959249 1594 raft_consensus.cc:1261] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260808 01:11:28.959745 1594 pending_rounds.cc:76] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78: Aborting all ops after (but not including) 5
I20260808 01:11:28.960052 1594 pending_rounds.cc:97] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 6.6
I20260808 01:11:28.816694 4961 raft_consensus.cc:2726] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:11:28.960276 1601 raft_consensus.cc:1261] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260808 01:11:28.959514 5095 raft_consensus.cc:2781] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:11:28.961081 4269 raft_consensus.cc:3037] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:28.961247 5095 raft_consensus.cc:686] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:28.962293 5095 consensus_queue.cc:237] T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:28.964826 5168 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:28.876906 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:28.966120 1519 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 9.
I20260808 01:11:28.813355 5140 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:28.855393 2209 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.821090 5088 raft_consensus.cc:3032] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:11:28.759557 5143 raft_consensus.cc:686] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:28.891650 1521 raft_consensus.cc:1226] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:28.894754 1596 raft_consensus.cc:1226] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 9. Current term is 10. Ops: []
W20260808 01:11:28.881173 4886 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.022s sys 0.025s
I20260808 01:11:28.810433 5089 consensus_queue.cc:260] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:28.880565 5034 raft_consensus.cc:2726] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 01:11:28.906286 5048 consensus_meta.cc:287] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.675s user 0.009s sys 0.003s
I20260808 01:11:28.956465 5166 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.789554 5000 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:28.958043 5062 consensus_queue.cc:260] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:28.959235 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 105454us
W20260808 01:11:28.990867 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 90605.3us
I20260808 01:11:29.006958 1410 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:28.856396 1604 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 10.
I20260808 01:11:28.960302 1594 raft_consensus.cc:2894] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Aborting config change with OpId 6.6: Aborted: Op aborted by new leader
I20260808 01:11:29.007941 1594 raft_consensus.cc:2866] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:28.959128 2202 raft_consensus.cc:1261] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:11:28.974761 5088 raft_consensus.cc:729] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:28.974363 4960 consensus_queue.cc:260] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:28.955413 5110 consensus_queue.cc:260] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:28.979202 5138 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:28.967278 5032 raft_consensus.cc:2726] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260808 01:11:28.994632 4886 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s user 0.022s sys 0.025s
I20260808 01:11:29.005630 1596 raft_consensus.cc:1226] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:28.991003 5020 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.009312 1604 raft_consensus.cc:1226] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:29.011646 2201 raft_consensus.cc:1226] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260808 01:11:29.013199 1410 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:29.018587 1594 pending_rounds.cc:97] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20260808 01:11:29.033133 1519 raft_consensus.cc:1226] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 6. Current term is 7. Ops: []
W20260808 01:11:29.034039 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 339871us
I20260808 01:11:29.041920 2209 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:29.042042 1594 raft_consensus.cc:2866] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:28.998464 5019 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:29.010428 5048 raft_consensus.cc:2726] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:29.010977 5000 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:29.018057 4887 raft_consensus.cc:2726] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:28.965382 5168 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:29.002573 5137 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:29.063645 4886 raft_consensus.cc:2781] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:11:29.081041 4886 raft_consensus.cc:686] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:29.059468 2202 raft_consensus.cc:1226] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:29.085439 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:29.086290 2201 raft_consensus.cc:2379] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 10: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:28.980804 5140 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:29.069278 5110 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:29.088666 5140 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:29.065241 4757 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:29.068841 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:29.032735 5053 consensus_queue.cc:1035] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260808 01:11:29.093693 1518 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 8.
I20260808 01:11:29.041539 5101 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:29.077409 5166 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:29.059455 5062 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:29.002718 4553 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:29.060400 2172 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:29.111616 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 51305.4us
W20260808 01:11:29.088543 5157 log.cc:924] Time spent T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20260808 01:11:29.041504 5086 consensus_queue.cc:1046] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
W20260808 01:11:29.176208 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 63758.6us
I20260808 01:11:29.043923 5138 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.072163 5098 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:29.090862 5020 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:29.088577 5031 raft_consensus.cc:2726] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:29.166455 5116 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:29.181166 2172 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:29.059193 5019 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.181348 5088 consensus_queue.cc:260] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.217343 2200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 802) took 1124 ms. Trace:
I20260808 01:11:29.198163 4733 raft_consensus.cc:2726] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260808 01:11:29.216902 5167 log.cc:924] Time spent T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20260808 01:11:29.218977 2200 rpcz_store.cc:276] 0808 01:11:28.092457 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:28.092979 (+ 522us) service_pool.cc:227] Handling call
0808 01:11:28.134242 (+ 41263us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:28.134689 (+ 447us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:28.134727 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:28.148180 (+ 13453us) log.cc:842] Serialized 26 byte log entry
0808 01:11:28.669890 (+521710us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:28.992277 (+322387us) log.cc:842] Serialized 14 byte log entry
0808 01:11:28.992670 (+ 393us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:28.992768 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:29.215886 (+223118us) raft_consensus.cc:1671] finished
0808 01:11:29.215910 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:29.217276 (+ 1366us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4083,"prepare.queue_time_us":298014,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":6915,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":26543,"spinlock_wait_cycles":3328,"thread_start_us":797048,"threads_started":2,"wal-append.queue_time_us":728700}
I20260808 01:11:29.217308 5097 raft_consensus.cc:2726] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:29.173497 5124 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:29.105558 4757 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:29.210428 5147 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:29.212365 5119 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:29.238045 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 224932us
I20260808 01:11:29.246747 1410 leader_election.cc:302] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:29.186985 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 221947us
I20260808 01:11:29.148447 5101 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.096891 5143 consensus_queue.cc:237] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.256058 2207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1182) took 1186 ms. Trace:
I20260808 01:11:29.207122 5087 consensus_queue.cc:1035] T 3f18b5317cac4512ad16a64664fab357 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260808 01:11:29.256407 2172 leader_election.cc:302] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:29.224025 4553 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:29.268440 2203 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1183) took 1016 ms. Trace:
I20260808 01:11:29.269102 1523 raft_consensus.cc:3037] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:29.260156 2207 rpcz_store.cc:276] 0808 01:11:28.069325 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:28.069508 (+ 183us) service_pool.cc:227] Handling call
0808 01:11:28.119536 (+ 50028us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:28.119815 (+ 279us) raft_consensus.cc:1520] Early marking committed up to index 6
0808 01:11:28.828761 (+708946us) log.cc:842] Serialized 14 byte log entry
0808 01:11:29.255703 (+426942us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:29.255753 (+ 50us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:29.255806 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:29.255886 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:29.255990 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7253,"prepare.queue_time_us":823477,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":7680,"thread_start_us":1091678,"threads_started":2,"wal-append.queue_time_us":411768}
I20260808 01:11:29.271628 2209 raft_consensus.cc:1226] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:29.251782 5053 raft_consensus.cc:2781] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:29.268904 2203 rpcz_store.cc:276] 0808 01:11:28.279963 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:28.649189 (+369226us) service_pool.cc:227] Handling call
0808 01:11:28.649490 (+ 301us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:28.649893 (+ 403us) raft_consensus.cc:1520] Early marking committed up to index 7
0808 01:11:28.982528 (+332635us) log.cc:842] Serialized 14 byte log entry
0808 01:11:29.268084 (+285556us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:29.268142 (+ 58us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:29.268192 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:29.268258 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:29.268380 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8762,"prepare.queue_time_us":293689,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"thread_start_us":592439,"threads_started":2}
I20260808 01:11:29.251988 5020 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:29.276175 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:29.280269 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:29.280912 1447 raft_consensus.cc:3037] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:29.281817 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:29.282711 1517 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 8.
I20260808 01:11:29.220414 5116 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:29.306442 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:29.268230 5168 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:29.276715 5127 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:29.258530 5097 consensus_queue.cc:1046] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
W20260808 01:11:29.312278 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 64058.8us
I20260808 01:11:29.321183 1446 raft_consensus.cc:2379] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 8: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:29.322851 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:29.287249 5172 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:29.285116 5065 raft_consensus.cc:2781] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:11:29.340310 1410 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:29.340483 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:29.251204 5119 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:29.343472 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: true dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:29.237457 4886 consensus_queue.cc:237] T bc9cccb55e8444778a9b2833e3d89a5a P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.299029 5094 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:29.265232 5044 consensus_queue.cc:1035] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260808 01:11:29.421435 1487 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:29.401736 1447 raft_consensus.cc:2454] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:29.261508 4961 consensus_queue.cc:1046] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20260808 01:11:29.267797 4553 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.403628 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:29.426026 1487 leader_election.cc:302] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
W20260808 01:11:29.321301 5099 consensus_meta.cc:287] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.502s user 0.001s sys 0.010s
I20260808 01:11:29.298535 5185 raft_consensus.cc:482] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:29.414148 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:29.414505 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:29.414961 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:29.284555 5137 consensus_queue.cc:1046] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.060s
I20260808 01:11:29.288971 5020 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:29.425012 2172 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:29.431752 5168 raft_consensus.cc:2781] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:29.431882 5065 raft_consensus.cc:686] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 9 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:29.402691 5053 raft_consensus.cc:686] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:29.336685 5127 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.512471 1601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 678) took 1467 ms. Trace:
I20260808 01:11:29.499182 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 1068) took 1199 ms. Trace:
W20260808 01:11:29.608286 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 183348us
I20260808 01:11:29.605427 1516 raft_consensus.cc:1261] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:11:29.492082 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:29.498340 1524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2216) took 1297 ms. Trace:
W20260808 01:11:29.591967 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 251755us
I20260808 01:11:29.610968 1524 rpcz_store.cc:276] 0808 01:11:28.200635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:28.282228 (+ 81593us) service_pool.cc:227] Handling call
0808 01:11:28.732223 (+449995us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:28.787526 (+ 55303us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:29.498124 (+710598us) spinlock_profiling.cc:243] Waited 449 ms on lock 0x7b54000ea450. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:29.498268 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1033825152}
I20260808 01:11:29.417996 5124 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:29.605212 2202 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:29.617388 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:29.446170 5147 raft_consensus.cc:2781] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:29.607486 2206 rpcz_store.cc:276] 0808 01:11:28.299516 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:28.694651 (+395135us) service_pool.cc:227] Handling call
0808 01:11:28.718197 (+ 23546us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:28.933627 (+215430us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:28.933659 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:28.933702 (+ 43us) raft_consensus.cc:1631] Marking committed up to 10
0808 01:11:28.933731 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:29.498964 (+565233us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b54003656d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:29.499003 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:29.499123 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":494917504}
I20260808 01:11:29.619589 1565 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
W20260808 01:11:29.494961 5173 log.cc:924] Time spent T 704a8e4165cc49079ada0c7993c21187 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.232s user 0.000s sys 0.001s
W20260808 01:11:29.489343 5183 log.cc:924] Time spent T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20260808 01:11:29.489343 5175 log.cc:924] Time spent T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.224s user 0.000s sys 0.000s
W20260808 01:11:29.493467 4568 consensus_meta.cc:287] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.267s user 0.009s sys 0.004s
W20260808 01:11:29.491544 5182 log.cc:924] Time spent T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
W20260808 01:11:29.492178 5193 log.cc:924] Time spent T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
I20260808 01:11:29.490844 5119 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:29.667569 5199 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:29.668220 1599 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:29.628270 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:29.488238 4757 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:29.505719 5047 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:29.607944 5019 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:29.427615 5094 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.456082 5138 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:29.664901 5191 log.cc:924] Time spent T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.056s user 0.001s sys 0.001s
I20260808 01:11:29.665805 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:29.408169 5172 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.666582 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:29.606791 1601 rpcz_store.cc:276] 0808 01:11:28.046289 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:28.858013 (+811724us) service_pool.cc:227] Handling call
0808 01:11:28.959919 (+101906us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:28.960655 (+ 736us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:29.512185 (+551530us) spinlock_profiling.cc:243] Waited 71.8 ms on lock 0x7b54002ad0d0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:29.512390 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":165159680}
I20260808 01:11:29.669534 1524 raft_consensus.cc:2454] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:29.606252 5208 raft_consensus.cc:2726] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:11:29.646001 5147 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:29.646114 5124 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.448557 5099 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.682051 2207 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 8.
I20260808 01:11:29.666163 5020 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:29.700913 2206 raft_consensus.cc:2379] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:29.654366 5101 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:29.706414 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:29.704870 1567 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:29.512422 5204 consensus_queue.cc:1046] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.172s
I20260808 01:11:29.593640 5179 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:29.592674 4553 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:29.669466 4961 raft_consensus.cc:2781] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:11:29.610204 5168 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:29.672276 5086 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:29.651400 5068 consensus_queue.cc:1046] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260808 01:11:29.746745 2202 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:29.685858 5200 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:29.750516 5068 raft_consensus.cc:2781] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:29.751201 5068 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:29.751559 5068 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:29.751964 5174 raft_consensus.cc:2781] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:29.745148 4961 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:29.754822 2205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 1069) took 1392 ms. Trace:
I20260808 01:11:29.588038 5116 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:11:29.740528 5000 consensus_meta.cc:287] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.630s user 0.011s sys 0.000s
I20260808 01:11:29.701992 5207 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:29.755311 2205 rpcz_store.cc:276] 0808 01:11:28.362944 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:28.686182 (+323238us) service_pool.cc:227] Handling call
0808 01:11:28.686555 (+ 373us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:28.690144 (+ 3589us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:28.690173 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:28.690608 (+ 435us) log.cc:842] Serialized 26 byte log entry
0808 01:11:29.099742 (+409134us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:29.099798 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:29.099905 (+ 107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:29.754557 (+654652us) raft_consensus.cc:1671] finished
0808 01:11:29.754582 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:29.754746 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24609,"raft.queue_time_us":3855,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":8603,"thread_start_us":364044,"threads_started":1,"wal-append.queue_time_us":516551}
I20260808 01:11:29.621001 5185 raft_consensus.cc:504] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.675017 5047 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.668174 5199 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.859767 1487 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
W20260808 01:11:29.784719 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 173708us
I20260808 01:11:29.845498 5189 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:29.874503 1523 consensus_meta.cc:287] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.605s user 0.012s sys 0.001s
I20260808 01:11:29.758486 5168 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:29.877851 1523 raft_consensus.cc:1261] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:11:29.880090 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:29.851373 5103 consensus_meta.cc:287] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.899s user 0.008s sys 0.005s
I20260808 01:11:29.882609 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:29.759317 4961 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:11:29.811479 5194 log.cc:924] Time spent T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.261s user 0.000s sys 0.001s
I20260808 01:11:29.887749 1520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:58250 (request call id 1099) took 1404 ms. Trace:
I20260808 01:11:29.884212 1598 raft_consensus.cc:1226] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 7. Current term is 8. Ops: []
I20260808 01:11:29.883333 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:29.883196 1440 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:29.710026 5147 raft_consensus.cc:3037] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:11:29.807585 5192 log.cc:924] Time spent T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.200s user 0.000s sys 0.001s
I20260808 01:11:29.900599 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:29.901538 1524 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:29.904274 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:29.770694 5019 raft_consensus.cc:3032] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:29.888195 1520 rpcz_store.cc:276] 0808 01:11:28.478142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:28.560483 (+ 82341us) service_pool.cc:227] Handling call
0808 01:11:29.875819 (+1315336us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b54000ea454. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca457c5 00007fe34ca3dc1a 00007fe3519816d9 00007fe34b06b184 00007fe34b06b112 00007fe34b06b041 00007fe34b06afbc 00007fe34b06a272 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:29.881926 (+ 6107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":491445120}
I20260808 01:11:29.891042 2201 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 11.
I20260808 01:11:29.906740 1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2397) took 1838 ms. Trace:
I20260808 01:11:29.905205 1524 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 7.
I20260808 01:11:29.814395 5172 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:29.879069 4553 raft_consensus.cc:2781] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:11:29.759034 5086 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:29.916913 1328 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet a83a963773094d6db6495b7009fd8690 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6b09dbefd24b4d52b385c4dff5697dc5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } } }
I20260808 01:11:29.762548 5179 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:29.920816 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 299707us
I20260808 01:11:29.921927 2207 raft_consensus.cc:1261] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:11:29.920285 2172 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:29.885141 5174 raft_consensus.cc:686] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:29.915433 4553 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:29.936722 4553 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:29.893285 5185 leader_election.cc:288] T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:29.937058 5174 consensus_queue.cc:237] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:29.938282 5210 log.cc:924] Time spent T 55fb799d94774eb18fed4dd8fdb5ac63 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260808 01:11:29.887811 5127 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:29.890187 5031 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:29.890235 4887 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:29.861941 5000 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.930233 5047 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:29.930387 5141 consensus_meta.cc:287] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 1.246s user 0.011s sys 0.003s
W20260808 01:11:29.930485 5201 log.cc:924] Time spent T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.307s user 0.001s sys 0.000s
I20260808 01:11:29.763726 5200 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.925271 5074 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:29.945890 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:29.894160 5214 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:29.885391 5207 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:29.967948 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2611aecd26485e8912ab55890ec0fa" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:29.968742 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 263962us
I20260808 01:11:29.907004 5138 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:11:29.972334 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 52167.4us
I20260808 01:11:29.930096 5124 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:30.000500 1444 raft_consensus.cc:1226] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:30.000823 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:29.972772 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 125752us
I20260808 01:11:29.909432 5019 raft_consensus.cc:729] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:29.988785 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:29.965536 4269 consensus_meta.cc:287] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.004s user 0.007s sys 0.007s
I20260808 01:11:29.989435 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
W20260808 01:11:29.965559 5191 log.cc:924] Time spent T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260808 01:11:29.990026 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:29.990444 1442 raft_consensus.cc:1226] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 11 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 10. Current term is 11. Ops: []
I20260808 01:11:29.991489 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:30.035800 1438 raft_consensus.cc:2379] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 6: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:29.768416 5053 consensus_queue.cc:237] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:29.991498 2203 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:29.992463 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:29.978155 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 78840.8us
I20260808 01:11:30.036973 1443 raft_consensus.cc:2379] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 6: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:29.995963 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:30.038816 1440 raft_consensus.cc:2454] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:29.965313 5094 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:29.971160 5181 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:30.010701 5110 consensus_meta.cc:287] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.922s user 0.006s sys 0.006s
I20260808 01:11:30.050976 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:30.013028 1445 raft_consensus.cc:2454] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 12.
I20260808 01:11:30.010710 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:29.883554 5189 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:30.064697 1408 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:30.030620 1490 leader_election.cc:302] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:30.052299 1520 raft_consensus.cc:2362] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 10. Re-sending same reply.
I20260808 01:11:29.976437 4887 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:30.031109 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:30.048516 2171 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:30.066672 2171 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:29.986805 5095 consensus_queue.cc:1035] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20260808 01:11:29.999815 5103 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:30.069437 1448 raft_consensus.cc:2379] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:30.054134 2205 raft_consensus.cc:1226] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:29.991643 5216 consensus_queue.cc:1035] T 7a2611aecd26485e8912ab55890ec0fa P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.076s
I20260808 01:11:30.032650 4886 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:30.063334 5199 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:11:30.063638 5205 log.cc:924] Time spent T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20260808 01:11:29.996593 5074 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.002750 5138 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:29.777058 5065 consensus_queue.cc:237] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.052362 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:30.013039 5049 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:30.098210 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:30.051795 5062 consensus_meta.cc:287] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.884s user 0.002s sys 0.011s
I20260808 01:11:29.976058 5031 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:30.078696 1447 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
W20260808 01:11:30.050335 5037 consensus_meta.cc:287] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 1.174s user 0.012s sys 0.002s
I20260808 01:11:30.078866 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:30.079936 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:30.109733 1519 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:30.109979 1596 raft_consensus.cc:2379] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 10: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:30.081344 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:30.085194 5215 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:30.110652 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:30.086082 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:30.013159 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:29.993831 5141 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.112178 1521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2229) took 1080 ms. Trace:
I20260808 01:11:30.073731 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:30.099009 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:30.105502 1440 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:30.116246 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:30.093385 5199 raft_consensus.cc:2781] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:30.117002 1519 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:30.013458 5208 consensus_queue.cc:1035] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.094s
I20260808 01:11:30.135250 5122 raft_consensus.cc:3032] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 8 LEADER]: Stepping down as leader of term 8
W20260808 01:11:30.136292 5222 log.cc:924] Time spent T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
I20260808 01:11:30.112568 1521 rpcz_store.cc:276] 0808 01:11:29.031382 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:29.031686 (+ 304us) service_pool.cc:227] Handling call
0808 01:11:29.032076 (+ 390us) raft_consensus.cc:1457] Updating replica for 2 ops
0808 01:11:29.032437 (+ 361us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:29.032465 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0808 01:11:29.046557 (+ 14092us) log.cc:842] Serialized 52 byte log entry
0808 01:11:29.254378 (+207821us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:29.260720 (+ 6342us) log.cc:842] Serialized 14 byte log entry
0808 01:11:29.263638 (+ 2918us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:29.263712 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:29.579073 (+315361us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x7b4800250758. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe3518e67f1 00007fe3518dd8e7 00007fe3518dd77a 00007fe3519d195f 00007fe3519d1627 00007fe3519e5b2c 00007fe3519e5ad8 00007fe3519e5a58 00007fe3519e5a13 00007fe3519e4cbc 00007fe350966be6
0808 01:11:30.102791 (+523718us) raft_consensus.cc:1671] finished
0808 01:11:30.111966 (+ 9175us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:30.112116 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2606,"prepare.queue_time_us":5995,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":8996,"raft.run_cpu_time_us":2655,"raft.run_wall_time_us":337946,"spinlock_wait_cycles":5584768,"thread_start_us":164673,"threads_started":1,"wal-append.queue_time_us":283167}
I20260808 01:11:30.130502 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:30.102872 5172 raft_consensus.cc:2781] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:30.064566 5181 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:30.140770 5200 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:30.083806 4886 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:30.141949 5181 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:30.143494 4886 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:30.146080 5168 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.117954 5037 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.149432 2208 raft_consensus.cc:1226] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 8. Current term is 9. Ops: []
I20260808 01:11:30.111776 5143 raft_consensus.cc:2726] T 7a2611aecd26485e8912ab55890ec0fa P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:30.133589 5228 raft_consensus.cc:2781] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:30.133675 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:30.134581 5179 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:30.134480 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:29.999989 5214 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.152449 1520 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:30.135337 4887 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:30.135241 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:30.157284 1523 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:30.136247 5122 raft_consensus.cc:729] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:30.079139 5103 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.115073 5090 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:30.136113 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:30.127336 1410 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:30.103539 5099 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:30.139896 5172 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:30.112427 5204 raft_consensus.cc:2781] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:30.134092 5207 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:30.138695 1522 raft_consensus.cc:1226] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:30.106313 5049 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:30.130069 5215 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:30.083645 5019 consensus_queue.cc:260] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:30.194983 2205 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260808 01:11:30.247633 1596 raft_consensus.cc:2454] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:30.247179 1516 raft_consensus.cc:1226] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 7. Current term is 8. Ops: []
I20260808 01:11:30.249135 1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:45612 (request call id 778) took 2084 ms. Trace:
I20260808 01:11:30.249626 1604 rpcz_store.cc:276] 0808 01:11:28.161754 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:29.038373 (+876619us) service_pool.cc:227] Handling call
0808 01:11:30.245519 (+1207146us) spinlock_profiling.cc:243] Waited 37.3 ms on lock 0x7b5400121954. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4af0 00007fe34ca47f2a 00007fe351984c6e 00007fe34b07b807 00007fe34b07b792 00007fe34b07b6c1 00007fe34b07b63c 00007fe34b07a8f2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:30.246278 (+ 759us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":85804288}
I20260808 01:11:30.189688 1523 raft_consensus.cc:1226] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 8. Current term is 9. Ops: [6.7-8.8]
W20260808 01:11:30.253423 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 67677us
I20260808 01:11:30.246922 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:30.129331 5199 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:30.255465 1601 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:30.258665 1602 raft_consensus.cc:2454] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:30.130825 5074 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:30.162228 5122 consensus_queue.cc:260] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.297060 1601 raft_consensus.cc:1226] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:30.263571 1604 raft_consensus.cc:1261] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260808 01:11:30.312470 1601 raft_consensus.cc:1226] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260808 01:11:30.264576 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 137351us
I20260808 01:11:30.219167 5213 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:30.141973 5189 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:30.200974 5214 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:30.321051 1568 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
W20260808 01:11:30.335197 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 245198us
W20260808 01:11:30.335956 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 1.73599e+06us
I20260808 01:11:30.336612 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:30.337491 1601 raft_consensus.cc:2396] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260808 01:11:30.218397 5141 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:30.332182 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 74477.7us
W20260808 01:11:30.332259 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 301707us
I20260808 01:11:30.347623 4731 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:30.348385 4731 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:30.349673 1523 raft_consensus.cc:1261] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:11:30.350054 5219 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:30.350721 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:30.350566 5219 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:30.352144 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:30.353262 1520 raft_consensus.cc:2396] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260808 01:11:30.354278 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:30.253847 5172 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:30.181381 5037 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:30.218634 5000 leader_election.cc:288] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:30.338172 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:30.338829 1444 raft_consensus.cc:1226] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260808 01:11:30.356115 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:30.295619 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 229032us
I20260808 01:11:30.340217 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:30.356688 1519 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:30.221578 4596 raft_consensus.cc:3032] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:11:30.340730 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:30.261168 5199 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:30.342481 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:30.359347 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 12 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:30.344285 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:30.351379 1523 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 4.
I20260808 01:11:30.351462 1521 raft_consensus.cc:1261] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:11:30.355646 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:30.361295 1447 raft_consensus.cc:1226] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 LEADER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:30.344323 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:30.363389 2207 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:30.265425 5020 raft_consensus.cc:2726] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:11:30.199623 5068 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:30.298359 4891 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:30.265654 5116 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:30.360174 1601 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:30.262245 5135 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:30.375190 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad0577278e046ad8387388b9e5e98a0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:30.329982 5143 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:30.377328 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:30.350510 4731 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:30.387485 1567 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:30.379758 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:30.174562 5147 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.298662 4553 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.397599 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:30.382055 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:30.384896 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:30.388447 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:30.358695 4596 raft_consensus.cc:729] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:30.354511 5219 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:30.364530 5074 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:30.394304 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:30.394199 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:30.260999 5090 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.342984 4886 consensus_queue.cc:1035] T 43bc87d4fdb1436f875364cc78ec3fca P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260808 01:11:30.368283 5168 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:30.254854 5086 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:30.403790 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:30.424553 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:30.385895 1490 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:30.403012 1568 leader_election.cc:302] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:30.426524 1568 leader_election.cc:302] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:30.381801 4269 raft_consensus.cc:482] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:30.427821 4269 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.430413 4269 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:30.431047 4269 raft_consensus.cc:2726] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:11:30.371097 5110 consensus_queue.cc:1046] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260808 01:11:30.415305 1519 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:30.424647 1601 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:30.433708 5218 consensus_queue.cc:1046] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.020s
I20260808 01:11:30.398875 1444 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 8.
I20260808 01:11:30.431608 4269 raft_consensus.cc:2726] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:11:30.435858 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 98355.7us
I20260808 01:11:30.392567 5038 consensus_queue.cc:1046] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260808 01:11:30.456044 5049 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:30.384320 5135 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:30.396585 4731 consensus_queue.cc:1035] T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20260808 01:11:30.391842 5143 raft_consensus.cc:504] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.414173 5138 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:30.439803 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:30.469815 1446 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:30.417438 5087 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:30.424113 5074 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.441555 5168 raft_consensus.cc:2781] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:30.441854 1440 raft_consensus.cc:1226] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 6. Current term is 7. Ops: []
W20260808 01:11:30.436405 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 76691.8us
I20260808 01:11:30.438021 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:30.437358 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 51574.9us
I20260808 01:11:30.478964 1523 raft_consensus.cc:1261] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:11:30.382781 5116 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.423269 5219 raft_consensus.cc:2726] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:30.392379 5034 consensus_queue.cc:1046] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260808 01:11:30.465499 1410 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:30.460014 5160 log.cc:924] Time spent T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 1.390s user 0.001s sys 0.004s
W20260808 01:11:30.471709 1567 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 84309.3us
I20260808 01:11:30.457942 5236 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:30.387892 5213 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:30.489661 1444 raft_consensus.cc:1261] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:11:30.477982 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:30.420305 5237 consensus_queue.cc:1046] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.071s
I20260808 01:11:30.470279 5084 raft_consensus.cc:2726] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:11:30.460467 5234 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:30.485523 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 59114.4us
I20260808 01:11:30.471132 5103 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:30.476666 5168 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:30.488524 2171 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:30.495532 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:30.497573 1568 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:30.502888 1601 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:30.476428 5087 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.487572 1410 leader_election.cc:302] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:30.484179 5233 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:30.507747 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:30.482342 5138 consensus_queue.cc:1046] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.010s
I20260808 01:11:30.513047 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:30.509256 1410 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:30.493542 5101 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:11:30.506497 5065 consensus_queue.cc:1035] T 3ad0577278e046ad8387388b9e5e98a0 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:11:30.516309 1567 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:30.514803 5116 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:30.506628 5233 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.517225 5060 consensus_queue.cc:1046] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20260808 01:11:30.512463 5207 raft_consensus.cc:2726] T 0108cd20b4bb4957b8a59eebd68479e4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:30.539868 5189 raft_consensus.cc:2781] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:30.539354 4731 consensus_queue.cc:1046] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260808 01:11:30.540906 4269 raft_consensus.cc:2726] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:11:30.541831 5034 raft_consensus.cc:482] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:30.539083 1568 leader_election.cc:302] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:30.500512 5234 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.499015 5236 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.516186 2207 raft_consensus.cc:2454] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:30.546455 1438 raft_consensus.cc:2454] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:30.501674 5068 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:30.548676 1517 raft_consensus.cc:1226] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:30.504400 5168 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:30.508307 5037 raft_consensus.cc:2726] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:11:30.520634 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:30.551981 1444 raft_consensus.cc:1226] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:30.551647 5241 consensus_queue.cc:1046] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
I20260808 01:11:30.552963 1525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2212) took 2359 ms. Trace:
I20260808 01:11:30.556602 1410 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:30.504786 5214 consensus_queue.cc:1035] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260808 01:11:30.654731 1521 raft_consensus.cc:3032] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:11:30.655179 1521 raft_consensus.cc:729] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:30.655428 1408 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:30.546422 5049 raft_consensus.cc:3032] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:30.654371 1601 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 8.
I20260808 01:11:30.591681 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:30.655997 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 108221us
I20260808 01:11:30.662029 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:30.557653 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:30.656415 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 99910.1us
I20260808 01:11:30.656512 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:30.618341 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:30.664942 1447 raft_consensus.cc:2362] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6. Re-sending same reply.
I20260808 01:11:30.664323 1410 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:30.665122 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:30.663549 1601 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 9.
I20260808 01:11:30.604538 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 1071) took 2009 ms. Trace:
I20260808 01:11:30.666087 1520 raft_consensus.cc:2396] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260808 01:11:30.602686 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1164) took 3763 ms. Trace:
I20260808 01:11:30.666286 2208 rpcz_store.cc:276] 0808 01:11:28.593986 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:28.727738 (+133752us) service_pool.cc:227] Handling call
0808 01:11:28.728120 (+ 382us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:30.149834 (+1421714us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:30.603330 (+453496us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54007d05d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:30.603501 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3268173184}
I20260808 01:11:30.578783 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:30.555573 4961 consensus_meta.cc:287] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.646s user 0.012s sys 0.000s
W20260808 01:11:30.602754 5226 log.cc:924] Time spent T 3ad0577278e046ad8387388b9e5e98a0 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.164s user 0.000s sys 0.001s
I20260808 01:11:30.660883 1446 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 7.
W20260808 01:11:30.657728 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 121445us
I20260808 01:11:30.681085 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:30.653911 5189 raft_consensus.cc:482] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:30.650220 5135 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:30.654027 1525 rpcz_store.cc:276] 0808 01:11:28.193581 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:28.193703 (+ 122us) service_pool.cc:227] Handling call
0808 01:11:28.194108 (+ 405us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:28.194542 (+ 434us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:28.194584 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:28.195189 (+ 605us) log.cc:842] Serialized 26 byte log entry
0808 01:11:28.676626 (+481437us) raft_consensus.cc:1631] Marking committed up to 3
0808 01:11:28.676663 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:28.679174 (+ 2511us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:30.423040 (+1743866us) spinlock_profiling.cc:243] Waited 4.15 ms on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:30.545228 (+122188us) raft_consensus.cc:1671] finished
0808 01:11:30.545252 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:30.552888 (+ 7636us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":39382,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"spinlock_wait_cycles":9533824,"thread_start_us":481031,"threads_started":1,"wal-append.queue_time_us":795246}
I20260808 01:11:30.640027 5216 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:30.684875 1410 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:30.591027 5038 raft_consensus.cc:2781] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:30.672605 5236 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:30.753697 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:30.913811 1447 raft_consensus.cc:2454] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:30.746860 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:30.746852 1446 raft_consensus.cc:3032] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:30.666893 2204 rpcz_store.cc:276] 0808 01:11:26.845091 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:26.845223 (+ 132us) service_pool.cc:227] Handling call
0808 01:11:28.364205 (+1518982us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:28.379744 (+ 15539us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:28.379778 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:28.945561 (+565783us) spinlock_profiling.cc:243] Waited 33 ms on lock 0x7b2c0002afa8. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34b422e11 00007fe34b432442 00007fe34ca7ac11 00007fe34ca3f58a 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772
0808 01:11:28.946022 (+ 461us) log.cc:842] Serialized 26 byte log entry
0808 01:11:29.232855 (+286833us) raft_consensus.cc:1631] Marking committed up to 4
0808 01:11:29.232910 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:29.232998 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:29.668383 (+435385us) raft_consensus.cc:1671] finished
0808 01:11:29.668404 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:30.602465 (+934061us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b5400708fd0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:30.602623 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15081,"raft.queue_time_us":15187,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":63,"spinlock_wait_cycles":3568367744,"thread_start_us":286398,"threads_started":1,"wal-append.queue_time_us":338272,"wal-append.run_cpu_time_us":11109,"wal-append.run_wall_time_us":1360622}
I20260808 01:11:30.915370 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:30.915944 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:30.600188 5200 consensus_queue.cc:1035] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.147s
I20260808 01:11:30.659510 5049 raft_consensus.cc:729] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:30.661435 5000 raft_consensus.cc:2726] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:30.658725 5116 raft_consensus.cc:2726] T 3ad0577278e046ad8387388b9e5e98a0 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:30.556473 5215 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:30.773607 1568 leader_election.cc:302] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:30.774389 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:30.617386 5213 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:30.664482 4922 raft_consensus.cc:2781] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:30.693733 5060 raft_consensus.cc:3032] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:11:30.614851 5143 leader_election.cc:288] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:30.903741 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 56369.3us
I20260808 01:11:30.909610 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:30.658991 1521 consensus_queue.cc:260] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.685259 5207 raft_consensus.cc:2726] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:30.916733 2204 raft_consensus.cc:2379] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 7: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:30.576754 4553 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:30.918955 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:30.914937 1446 raft_consensus.cc:729] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:30.677845 5234 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:30.921824 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 237071us
I20260808 01:11:30.673332 5233 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:30.777529 5189 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:30.945160 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:30.669051 5053 consensus_queue.cc:1046] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.284s
I20260808 01:11:30.665104 5034 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.760730 4961 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.007970 5065 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:31.017671 5060 raft_consensus.cc:729] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:31.008831 1521 raft_consensus.cc:3037] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:31.023330 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:31.054924 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:30.927366 4887 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:30.925978 5236 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:30.955024 5185 raft_consensus.cc:2781] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:31.061386 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:30.936949 5074 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.022334 1446 consensus_queue.cc:260] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.064428 1446 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:30.912554 5216 raft_consensus.cc:504] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:30.901449 5038 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:31.029484 4553 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:31.029145 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:31.079252 2207 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:31.012403 5213 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:30.918581 5199 consensus_meta.cc:287] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.548s user 0.011s sys 0.000s
I20260808 01:11:31.048120 1594 raft_consensus.cc:2354] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 8. Current term is 9.
I20260808 01:11:31.048564 4961 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:31.013549 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:31.021865 4922 raft_consensus.cc:686] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:31.024782 5143 raft_consensus.cc:482] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:31.042120 1410 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:31.067456 5185 raft_consensus.cc:482] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:31.049575 5207 consensus_queue.cc:1035] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 01:11:31.081759 5038 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:31.050348 5037 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:31.034497 5138 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:31.059883 5053 raft_consensus.cc:3032] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Stepping down as leader of term 8
W20260808 01:11:31.097690 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 55660.4us
I20260808 01:11:31.122655 1410 leader_election.cc:302] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:30.916524 5203 consensus_queue.cc:1046] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.115s
I20260808 01:11:31.068306 5218 raft_consensus.cc:2726] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:11:31.030743 4269 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:31.105250 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:31.111420 1521 raft_consensus.cc:1261] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20260808 01:11:31.140511 1521 pending_rounds.cc:76] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5: Aborting all ops after (but not including) 6
I20260808 01:11:31.140797 1521 pending_rounds.cc:97] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260808 01:11:31.141044 1521 raft_consensus.cc:2866] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:31.104511 5215 raft_consensus.cc:2781] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:31.029853 5087 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:31.105831 5185 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:31.059545 5065 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.137210 1490 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:31.044176 5116 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:31.050139 5049 consensus_queue.cc:260] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.160638 2204 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:31.162006 5065 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.108362 5060 consensus_queue.cc:260] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.163326 5060 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:11:31.167936 2234 heartbeater.cc:644] Failed to heartbeat to 127.31.212.126:33415 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.31.212.126:33415 timed out after 15.000s (SENT)
I20260808 01:11:31.168597 2204 raft_consensus.cc:2396] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260808 01:11:31.173223 5034 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:31.174775 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 166570us
I20260808 01:11:31.175524 1487 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:31.122189 5053 raft_consensus.cc:729] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:31.152889 1490 leader_election.cc:302] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:31.070163 4887 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.089632 4553 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.155107 5116 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.178153 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:31.094779 5213 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.149747 5215 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:31.069921 5236 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.173775 4922 consensus_queue.cc:237] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.146193 5203 consensus_queue.cc:1225] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer 3fb1a6380d27401a831cfe5f86a4cc9d log is divergent from this leader: its last log entry 7.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:11:31.203109 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:31.096107 5199 raft_consensus.cc:504] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.103936 5143 raft_consensus.cc:504] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.128377 5138 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.129379 5037 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.143626 4886 consensus_queue.cc:1035] T 45ac77bfbd0c4849bbda5cd2e754a927 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260808 01:11:31.203780 1520 raft_consensus.cc:2454] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
I20260808 01:11:31.207279 1525 raft_consensus.cc:1261] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260808 01:11:31.148675 4269 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.196476 5215 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:31.140969 5257 raft_consensus.cc:2781] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:31.203645 5065 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:31.248922 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 475502us
I20260808 01:11:31.246649 5257 raft_consensus.cc:482] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:31.238814 5053 consensus_queue.cc:260] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.247479 5259 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:31.259501 5053 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:31.240482 5241 consensus_queue.cc:1035] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260808 01:11:31.255882 5037 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.258163 5199 leader_election.cc:288] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.257166 5213 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:31.273216 5254 raft_consensus.cc:2781] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:31.287250 4887 leader_election.cc:288] T a83a963773094d6db6495b7009fd8690 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:31.256170 5257 raft_consensus.cc:3037] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:11:31.288579 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 135766us
W20260808 01:11:31.291320 2171 leader_election.cc:334] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 2.030s (SENT)
I20260808 01:11:31.253244 5263 raft_consensus.cc:2781] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:31.295451 2207 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 9.
I20260808 01:11:31.285121 5262 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:31.266997 5264 raft_consensus.cc:2726] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:31.299182 1446 raft_consensus.cc:1261] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260808 01:11:31.252599 5065 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.308977 1446 pending_rounds.cc:76] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d: Aborting all ops after (but not including) 6
I20260808 01:11:31.309525 1446 pending_rounds.cc:97] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260808 01:11:31.303112 1490 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:31.303049 2171 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:31.304003 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:31.310822 1520 raft_consensus.cc:2454] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:31.312019 1487 leader_election.cc:302] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:31.305930 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:31.288743 5216 leader_election.cc:288] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.309998 1446 raft_consensus.cc:2866] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:11:31.320161 4269 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:31.320753 5181 raft_consensus.cc:2781] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:31.321223 5181 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:31.320971 5189 raft_consensus.cc:504] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.321605 5181 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:31.298071 5254 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:31.314181 5270 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:31.326233 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.326785 2171 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:31.332746 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.332088 2171 leader_election.cc:302] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:31.335088 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:31.335819 2207 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Advancing to term 9
I20260808 01:11:31.333577 1443 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:31.337458 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:31.333909 1489 leader_election.cc:302] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:31.309327 5143 leader_election.cc:288] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:31.340315 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.341079 1443 raft_consensus.cc:2379] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 7: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:31.340109 5268 raft_consensus.cc:2781] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:11:31.344324 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.322431 5037 raft_consensus.cc:2726] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:11:31.322469 5252 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:31.329120 5254 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:31.332888 5241 raft_consensus.cc:2781] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:31.346915 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.361351 1440 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:31.319182 5138 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.348265 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.329486 5236 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:31.349069 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:31.365948 1443 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:31.315030 5263 raft_consensus.cc:686] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:31.319439 4953 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:31.342541 4891 consensus_meta.cc:287] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.961s user 0.014s sys 0.000s
I20260808 01:11:31.354502 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:31.340433 5216 raft_consensus.cc:2781] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:31.290452 5259 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.346547 5065 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.343004 5264 raft_consensus.cc:2726] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:11:31.376365 4553 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.377609 2171 leader_election.cc:302] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:31.311085 5262 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.328790 5087 consensus_queue.cc:1035] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260808 01:11:31.347317 5268 raft_consensus.cc:686] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:31.354684 5112 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:11:31.354489 5116 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:31.378998 4833 raft_consensus.cc:482] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:31.373425 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.383673 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:31.367947 5037 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:31.392299 5262 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.377187 5216 raft_consensus.cc:482] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:31.365760 5241 raft_consensus.cc:482] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:11:31.393638 5101 consensus_meta.cc:287] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.848s user 0.010s sys 0.001s
I20260808 01:11:31.364573 5252 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.403865 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:31.376858 4953 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.388033 5065 raft_consensus.cc:2726] T c12c84f631ee4ca7b405e28317db1b8a P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:31.412191 2207 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 9.
I20260808 01:11:31.405669 1410 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:31.417239 1520 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:31.417901 1568 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:31.397080 5189 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.394829 5199 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:31.428970 1568 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:31.434067 1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 672) took 3392 ms. Trace:
W20260808 01:11:31.417840 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 114798us
I20260808 01:11:31.434250 5268 consensus_queue.cc:237] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.406574 5216 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:31.338088 5270 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.424407 5101 raft_consensus.cc:2726] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:11:31.442677 5099 raft_consensus.cc:2726] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:31.428903 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:31.441557 1568 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters: 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:31.442380 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.438082 1603 rpcz_store.cc:276] 0808 01:11:28.041465 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:28.801661 (+760196us) service_pool.cc:227] Handling call
0808 01:11:28.802004 (+ 343us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:28.852279 (+ 50275us) raft_consensus.cc:1520] Early marking committed up to index 8
0808 01:11:29.219386 (+367107us) log.cc:842] Serialized 14 byte log entry
0808 01:11:29.418858 (+199472us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:29.418937 (+ 79us) raft_consensus.cc:1631] Marking committed up to 8
0808 01:11:29.418985 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:31.433801 (+2014816us) spinlock_profiling.cc:243] Waited 27.6 ms on lock 0x7b54002ae4d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:31.433853 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:31.433982 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":96965,"prepare.queue_time_us":394076,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":4655,"raft.queue_time_us":5861,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":11561,"spinlock_wait_cycles":63437184,"thread_start_us":481822,"threads_started":2,"wal-append.queue_time_us":356383,"wal-append.run_cpu_time_us":1299,"wal-append.run_wall_time_us":1016405}
I20260808 01:11:31.418140 5241 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:31.457463 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:31.463538 1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 688) took 1086 ms. Trace:
I20260808 01:11:31.397838 4833 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.464183 1603 rpcz_store.cc:276] 0808 01:11:30.376903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.455158 (+1078255us) service_pool.cc:227] Handling call
0808 01:11:31.463404 (+ 8246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:31.435207 5116 raft_consensus.cc:2781] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:31.401657 5037 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.437515 5213 raft_consensus.cc:2781] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:31.468645 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:31.474035 1603 raft_consensus.cc:2454] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 10.
I20260808 01:11:31.474654 1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 695) took 1091 ms. Trace:
W20260808 01:11:31.475723 5168 consensus_meta.cc:287] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.780s user 0.009s sys 0.002s
I20260808 01:11:31.475822 1603 rpcz_store.cc:276] 0808 01:11:30.383633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.468325 (+1084692us) service_pool.cc:227] Handling call
0808 01:11:31.474597 (+ 6272us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:31.479523 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:31.481388 1603 raft_consensus.cc:2362] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 10. Re-sending same reply.
I20260808 01:11:31.443728 5199 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.483517 1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 689) took 1105 ms. Trace:
W20260808 01:11:31.528143 1568 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 87187.2us
I20260808 01:11:31.528622 5273 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:31.378137 5263 consensus_queue.cc:237] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.503937 1604 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:31.523428 1520 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:31.471000 5116 raft_consensus.cc:482] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:31.538746 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:31.538887 2171 leader_election.cc:302] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:31.534466 4953 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:31.539171 5116 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:31.469290 5099 raft_consensus.cc:2781] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:31.537292 1490 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:31.535580 1603 rpcz_store.cc:276] 0808 01:11:30.378436 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.478080 (+1099644us) service_pool.cc:227] Handling call
0808 01:11:31.483459 (+ 5379us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:31.540851 1520 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 7.
I20260808 01:11:31.541692 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:31.541692 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:31.483482 5172 consensus_meta.cc:287] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 1.127s user 0.014s sys 0.000s
I20260808 01:11:31.543747 2171 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:31.545049 1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2709) took 1170 ms. Trace:
I20260808 01:11:31.545428 1603 rpcz_store.cc:276] 0808 01:11:30.374877 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.541302 (+1166425us) service_pool.cc:227] Handling call
0808 01:11:31.544991 (+ 3689us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:31.544680 4953 raft_consensus.cc:2726] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:31.477187 5213 raft_consensus.cc:482] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:31.558902 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:31.509130 5038 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.476303 5168 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.560313 1520 raft_consensus.cc:2379] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
W20260808 01:11:31.560261 1487 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 248306us
I20260808 01:11:31.560824 1522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2217) took 3359 ms. Trace:
I20260808 01:11:31.561401 1522 rpcz_store.cc:276] 0808 01:11:28.200974 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:28.278999 (+ 78025us) service_pool.cc:227] Handling call
0808 01:11:28.301646 (+ 22647us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:30.247731 (+1946085us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:31.560566 (+1312835us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b54000a78d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:31.560764 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4224344448}
I20260808 01:11:31.549588 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:31.564970 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:31.545246 5265 raft_consensus.cc:2781] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:31.575191 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:31.544780 5087 raft_consensus.cc:2781] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:31.576751 1603 raft_consensus.cc:3037] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:31.580066 5038 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:31.544214 5099 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:11:31.583751 2171 leader_election.cc:334] T 516db2422f0644aebfd1999735efdbb8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 2.184s (SENT)
I20260808 01:11:31.586256 1522 raft_consensus.cc:1261] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260808 01:11:31.576351 5278 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:31.564589 5185 raft_consensus.cc:504] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.590912 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:31.560757 5213 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:31.568353 5259 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.600131 5060 consensus_queue.cc:1035] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:31.602543 1439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 877) took 2323 ms. Trace:
I20260808 01:11:31.602941 1439 rpcz_store.cc:276] 0808 01:11:29.279558 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:29.281771 (+ 2213us) service_pool.cc:227] Handling call
0808 01:11:29.305994 (+ 24223us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:29.885952 (+579958us) raft_consensus.cc:1520] Early marking committed up to index 8
0808 01:11:29.886290 (+ 338us) log.cc:842] Serialized 14 byte log entry
0808 01:11:30.003693 (+117403us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:30.003742 (+ 49us) raft_consensus.cc:1631] Marking committed up to 8
0808 01:11:30.003793 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:31.602291 (+1598498us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b540030c0d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:31.602343 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:31.602482 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":205,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":76,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":25207,"spinlock_wait_cycles":1333270016,"thread_start_us":116962,"threads_started":1,"wal-append.queue_time_us":423604}
W20260808 01:11:31.589071 5276 log.cc:924] Time spent T a539e94f6f744f20b65ea3fa4bbf8be4 P a4862d0938c84fc4bd840e4c17acae78: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20260808 01:11:31.584281 5065 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:31.608933 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:31.583619 5087 raft_consensus.cc:482] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:31.548254 5172 raft_consensus.cc:504] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.616199 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.589277 5265 raft_consensus.cc:686] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:31.616719 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:31.587242 5099 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:31.581882 4269 consensus_queue.cc:1046] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
I20260808 01:11:31.623463 1410 leader_election.cc:302] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:31.624164 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:31.624845 1522 raft_consensus.cc:2454] T a83a963773094d6db6495b7009fd8690 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 9.
I20260808 01:11:31.633827 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821d5636d01e4067bac2bbd7475921ef" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.636950 2203 raft_consensus.cc:1226] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260808 01:11:31.639793 5263 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:31.641137 5263 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.640414 1603 raft_consensus.cc:2454] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 9.
I20260808 01:11:31.644869 1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 694) took 1264 ms. Trace:
I20260808 01:11:31.645401 1603 rpcz_store.cc:276] 0808 01:11:30.380892 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.574844 (+1193952us) service_pool.cc:227] Handling call
0808 01:11:31.644789 (+ 69945us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:31.646911 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:31.648032 1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2701) took 1275 ms. Trace:
I20260808 01:11:31.650045 1603 rpcz_store.cc:276] 0808 01:11:30.372560 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.646569 (+1274009us) service_pool.cc:227] Handling call
0808 01:11:31.647965 (+ 1396us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:31.651283 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:31.652138 1603 raft_consensus.cc:2362] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 8. Re-sending same reply.
I20260808 01:11:31.652849 1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 690) took 1274 ms. Trace:
I20260808 01:11:31.649534 5263 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.653302 1603 rpcz_store.cc:276] 0808 01:11:30.378717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.650975 (+1272258us) service_pool.cc:227] Handling call
0808 01:11:31.652787 (+ 1812us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:31.655077 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:31.656096 1603 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:31.657218 1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2706) took 1283 ms. Trace:
I20260808 01:11:31.657842 1603 rpcz_store.cc:276] 0808 01:11:30.373760 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.654769 (+1281009us) service_pool.cc:227] Handling call
0808 01:11:31.657159 (+ 2390us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:31.659114 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:31.659984 1603 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:31.660657 1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2705) took 1287 ms. Trace:
I20260808 01:11:31.661095 1603 rpcz_store.cc:276] 0808 01:11:30.373549 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.658770 (+1285221us) service_pool.cc:227] Handling call
0808 01:11:31.660598 (+ 1828us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:31.663069 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:31.665138 1489 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:31.667243 5263 raft_consensus.cc:2726] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:31.668486 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:31.669171 1522 raft_consensus.cc:3037] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 8
I20260808 01:11:31.626609 5101 raft_consensus.cc:2726] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:31.672574 5037 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.618685 5087 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:31.617381 5266 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:31.617955 5038 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
W20260808 01:11:31.683260 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 93535.9us
I20260808 01:11:31.677597 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:31.679677 1490 leader_election.cc:302] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:31.684387 1603 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:31.684932 1603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 692) took 1304 ms. Trace:
I20260808 01:11:31.687474 1603 rpcz_store.cc:276] 0808 01:11:30.380345 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.668111 (+1287766us) service_pool.cc:227] Handling call
0808 01:11:31.684883 (+ 16772us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:31.689112 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:31.690124 1603 raft_consensus.cc:2379] T ca9890c6b79a40fcbbf70e44142a95d0 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 6: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:31.691385 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:31.543556 5273 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.587963 5278 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:31.619665 5065 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.702134 1603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a963773094d6db6495b7009fd8690" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:31.704159 1443 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 9.
I20260808 01:11:31.694209 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.688273 5263 raft_consensus.cc:2726] T 821d5636d01e4067bac2bbd7475921ef P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:31.710865 5273 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:31.688135 5038 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.713105 5038 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:31.713495 5275 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:31.714275 5275 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.705806 1439 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:31.717805 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:31.718521 1522 raft_consensus.cc:2454] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 8.
I20260808 01:11:31.720065 1410 leader_election.cc:302] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:31.721823 4733 raft_consensus.cc:3032] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:11:31.722471 4733 raft_consensus.cc:729] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:31.718888 1443 raft_consensus.cc:2379] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 7: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:31.717361 5120 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:31.727454 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:31.718132 2172 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:31.729601 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:31.731551 1522 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:31.731112 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:31.697492 5252 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:31.725958 5120 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.733294 5273 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:31.738562 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:31.746042 1522 raft_consensus.cc:3037] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 7
I20260808 01:11:31.745990 1568 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:31.746685 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.747740 1439 raft_consensus.cc:2379] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 10: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:31.748526 5038 raft_consensus.cc:2781] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:11:31.720100 5199 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.749184 5038 raft_consensus.cc:686] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [term 9 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:31.730983 5101 raft_consensus.cc:2781] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:11:31.750172 5101 raft_consensus.cc:686] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:31.751385 5263 raft_consensus.cc:482] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:31.751323 5101 consensus_queue.cc:237] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.752419 5263 raft_consensus.cc:504] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.757365 5038 consensus_queue.cc:237] T d80a0f28c7c94d45b7983fc0b1386caf P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.738740 4733 consensus_queue.cc:260] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.762423 2208 raft_consensus.cc:1226] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 7. Current term is 8. Ops: []
I20260808 01:11:31.691886 5215 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.763072 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 827) took 1211 ms. Trace:
I20260808 01:11:31.713143 5270 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:31.766801 2204 rpcz_store.cc:276] 0808 01:11:30.552019 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.061091 (+509072us) service_pool.cc:227] Handling call
0808 01:11:31.759683 (+698592us) spinlock_profiling.cc:243] Waited 98.6 ms on lock 0x7b5400440f54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:31.762992 (+ 3309us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":226726784}
I20260808 01:11:31.768575 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:31.769943 5065 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.771081 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 826) took 1219 ms. Trace:
I20260808 01:11:31.771414 2204 rpcz_store.cc:276] 0808 01:11:30.551662 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.769947 (+1218285us) service_pool.cc:227] Handling call
0808 01:11:31.770282 (+ 335us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:31.770617 (+ 335us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:31.770656 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:31.770710 (+ 54us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:31.770880 (+ 170us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:31.770937 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:31.771033 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:31.772854 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 828) took 1220 ms. Trace:
I20260808 01:11:31.775887 2204 rpcz_store.cc:276] 0808 01:11:30.552323 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.772176 (+1219853us) service_pool.cc:227] Handling call
0808 01:11:31.772386 (+ 210us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:31.772578 (+ 192us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:31.772593 (+ 15us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:31.772618 (+ 25us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:31.772671 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:31.772718 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:31.772813 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:29.908798 1597 rpcz_store.cc:276] 0808 01:11:28.068064 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:28.938715 (+870651us) service_pool.cc:227] Handling call
0808 01:11:28.939125 (+ 410us) raft_consensus.cc:1457] Updating replica for 3 ops
0808 01:11:28.974606 (+ 35481us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:28.974643 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0808 01:11:28.979597 (+ 4954us) log.cc:842] Serialized 78 byte log entry
0808 01:11:29.313808 (+334211us) raft_consensus.cc:1631] Marking committed up to 9
0808 01:11:29.684895 (+371087us) log.cc:842] Serialized 14 byte log entry
0808 01:11:29.686331 (+ 1436us) log.cc:842] Serialized 14 byte log entry
0808 01:11:29.898675 (+212344us) log.cc:842] Serialized 14 byte log entry
0808 01:11:29.906059 (+ 7384us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:29.906432 (+ 373us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:29.906499 (+ 67us) raft_consensus.cc:1671] finished
0808 01:11:29.906516 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:29.906683 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":97211,"prepare.queue_time_us":100972,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":2764,"raft.queue_time_us":152739,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":147,"spinlock_wait_cycles":77193856,"thread_start_us":822961,"threads_started":3,"wal-append.queue_time_us":622319}
I20260808 01:11:31.786967 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 829) took 1225 ms. Trace:
I20260808 01:11:31.787585 2204 rpcz_store.cc:276] 0808 01:11:30.555869 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.776816 (+1220947us) service_pool.cc:227] Handling call
0808 01:11:31.777120 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:31.777307 (+ 187us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:31.777332 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:31.777378 (+ 46us) raft_consensus.cc:1631] Marking committed up to 9
0808 01:11:31.777418 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:31.777481 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:31.779748 (+ 2267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:31.761835 4733 raft_consensus.cc:3037] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:11:31.786973 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 100701us
I20260808 01:11:31.789400 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.806921 2171 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:31.767588 5168 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:31.797812 5215 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:31.804256 1410 leader_election.cc:334] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 1.805s (SENT)
I20260808 01:11:31.762400 5185 leader_election.cc:288] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.808703 2171 leader_election.cc:302] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:31.778976 5252 raft_consensus.cc:2781] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:31.813565 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:31.810215 2204 raft_consensus.cc:1226] T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd [term 5 LEADER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 4. Current term is 5. Ops: []
I20260808 01:11:31.812076 1410 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:31.814741 1443 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:31.815037 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1200) took 1157 ms. Trace:
I20260808 01:11:31.816097 2204 rpcz_store.cc:276] 0808 01:11:30.657482 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.809049 (+1151567us) service_pool.cc:227] Handling call
0808 01:11:31.810025 (+ 976us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:31.814797 (+ 4772us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:31.814977 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:31.818112 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:31.819902 1490 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:31.820824 2171 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:31.821648 1522 raft_consensus.cc:2454] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:31.822854 1410 leader_election.cc:302] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:31.823920 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:31.825666 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.826479 2171 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:31.800828 5241 raft_consensus.cc:504] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.767795 5273 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.831247 1565 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:31.812335 5282 raft_consensus.cc:2781] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:11:31.835260 5278 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.837574 4596 consensus_queue.cc:260] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.839929 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.843811 2201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49270 (request call id 1080) took 2203 ms. Trace:
I20260808 01:11:31.847074 2204 raft_consensus.cc:1261] T 1ccc57605ae24bf4bacee4e4741bbe95 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260808 01:11:31.847175 5285 raft_consensus.cc:2726] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:11:31.730674 5265 consensus_queue.cc:237] T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.816151 5065 raft_consensus.cc:2781] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:11:31.833122 5094 raft_consensus.cc:2781] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:31.827487 5185 raft_consensus.cc:2726] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:11:31.852332 5019 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:31.830260 5215 raft_consensus.cc:2781] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:31.886422 5089 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:31.881760 5094 raft_consensus.cc:482] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:31.888772 5189 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:31.889465 5094 raft_consensus.cc:3037] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:31.889863 5189 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:31.894930 5257 consensus_meta.cc:287] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.583s user 0.007s sys 0.011s
I20260808 01:11:31.857671 5145 leader_election.cc:288] T 0108cd20b4bb4957b8a59eebd68479e4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.828970 5258 raft_consensus.cc:2726] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:11:31.862185 5053 consensus_meta.cc:287] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.569s user 0.010s sys 0.005s
I20260808 01:11:31.825269 5168 raft_consensus.cc:2781] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:11:31.831595 5172 leader_election.cc:288] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:31.866415 5273 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.817131 5252 raft_consensus.cc:482] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:31.869534 2200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 1095) took 1454 ms. Trace:
I20260808 01:11:31.870114 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:31.872782 1445 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55042 (request call id 2523) took 1884 ms (client timeout 2327 ms). Trace:
I20260808 01:11:31.870921 2201 rpcz_store.cc:276] 0808 01:11:29.640555 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:29.673978 (+ 33423us) service_pool.cc:227] Handling call
0808 01:11:31.843596 (+2169618us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54005f05d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:31.843711 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":345397120}
I20260808 01:11:31.701645 5266 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.912750 5257 raft_consensus.cc:504] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.949954 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:31.896092 5263 leader_election.cc:288] T 93b19a284f2b41fe843e33e399ea8ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.951519 2201 raft_consensus.cc:2396] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 01:11:31.834393 5282 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:31.857038 5099 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.953248 5257 leader_election.cc:288] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:31.953667 5282 raft_consensus.cc:3037] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:11:31.918709 5215 raft_consensus.cc:686] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:31.900074 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.883798 5275 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:31.879205 4596 raft_consensus.cc:3037] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:31.955124 5168 raft_consensus.cc:686] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:31.955366 5215 consensus_queue.cc:237] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.956979 5168 consensus_queue.cc:237] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.958441 1597 raft_consensus.cc:2454] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:31.958904 1568 leader_election.cc:302] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:31.959175 5015 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:31.960237 5275 raft_consensus.cc:2726] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:11:31.961294 5248 raft_consensus.cc:3032] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:31.961907 5248 raft_consensus.cc:729] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:31.962513 5087 raft_consensus.cc:504] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.964192 5015 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.965157 5116 raft_consensus.cc:504] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.967103 5099 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:31.961905 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.969416 5266 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:31.970052 1443 raft_consensus.cc:2379] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 6: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:31.971185 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.876047 4733 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:31.972651 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:31.971923 1443 raft_consensus.cc:2454] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:31.973178 4733 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:31.974745 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.976078 5252 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:31.976351 5049 raft_consensus.cc:3037] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:31.977711 5090 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:31.946699 1408 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:31.945726 1439 raft_consensus.cc:2379] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 9: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:31.980422 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.978945 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.981086 1439 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 10.
I20260808 01:11:31.981150 1443 raft_consensus.cc:2454] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:31.981906 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1197) took 1430 ms. Trace:
I20260808 01:11:31.982393 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:31.982376 2208 rpcz_store.cc:276] 0808 01:11:30.551039 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.237732 (+686693us) service_pool.cc:227] Handling call
0808 01:11:31.762226 (+524494us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:31.763886 (+ 1660us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:31.981722 (+217836us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b5400440f50. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:31.981850 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1183547136}
I20260808 01:11:31.983013 1446 raft_consensus.cc:2379] T 87907e719e4c46c4a19015f3f3bba308 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 6: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:31.984100 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:31.985002 1443 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:11:31.986328 1599 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2697) took 3655 ms (client timeout 1779 ms). Trace:
W20260808 01:11:31.986711 1599 rpcz_store.cc:269] 0808 01:11:28.331373 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:28.676109 (+344736us) service_pool.cc:227] Handling call
0808 01:11:31.986135 (+3310026us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b54002dbed4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:31.986235 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2278032640}
I20260808 01:11:31.823674 5181 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.964076 5248 consensus_queue.cc:260] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:31.990219 5248 raft_consensus.cc:3037] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:31.991495 1598 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2693) took 3952 ms. Trace:
I20260808 01:11:31.945091 5053 raft_consensus.cc:482] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
W20260808 01:11:31.945611 1445 rpcz_store.cc:269] 0808 01:11:29.988294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.839508 (+1851214us) service_pool.cc:227] Handling call
0808 01:11:31.872718 (+ 33210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:31.944998 2200 rpcz_store.cc:276] 0808 01:11:30.415197 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:30.508322 (+ 93125us) service_pool.cc:227] Handling call
0808 01:11:30.545682 (+ 37360us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:30.552395 (+ 6713us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:30.552440 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:30.553254 (+ 814us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:30.553300 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:31.868230 (+1314930us) spinlock_profiling.cc:243] Waited 6.5 ms on lock 0x7b54004835d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:31.869338 (+ 1108us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:31.869475 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14954752}
I20260808 01:11:31.993918 1568 leader_election.cc:302] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:31.999191 1568 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:31.982152 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:31.994333 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:31.985129 1448 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60392 (request call id 887) took 1879 ms (client timeout 1769 ms). Trace:
I20260808 01:11:31.991967 1598 rpcz_store.cc:276] 0808 01:11:28.038723 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:28.752542 (+713819us) service_pool.cc:227] Handling call
0808 01:11:28.752823 (+ 281us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:29.890580 (+1137757us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:31.991201 (+2100621us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54001e9e54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:31.991409 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2601478400}
I20260808 01:11:31.995877 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:31.996555 1443 raft_consensus.cc:2454] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:32.001711 2208 raft_consensus.cc:2454] T b316a05b1f7448158ae76530ee0ee127 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:32.001931 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:32.002802 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:32.003571 1443 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 10.
I20260808 01:11:32.003381 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40a08ace944a88ad5bf8ba22f2a40" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:32.010202 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 121754us
W20260808 01:11:31.906862 5254 consensus_meta.cc:287] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.538s user 0.007s sys 0.011s
I20260808 01:11:32.016871 5254 raft_consensus.cc:504] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:32.021274 1522 raft_consensus.cc:2379] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 8: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
W20260808 01:11:32.000741 1448 rpcz_store.cc:269] 0808 01:11:30.105276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.984832 (+1879556us) service_pool.cc:214] Skipping call since client already timed out
0808 01:11:31.985007 (+ 175us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 01:11:32.004623 1598 raft_consensus.cc:2454] T 44f40a08ace944a88ad5bf8ba22f2a40 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 9.
I20260808 01:11:32.029047 1448 raft_consensus.cc:1226] T 93b19a284f2b41fe843e33e399ea8ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:32.036723 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:32.037639 2171 leader_election.cc:302] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:32.034911 1565 leader_election.cc:302] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:32.038859 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:32.037147 1568 leader_election.cc:302] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:32.035457 2207 raft_consensus.cc:2379] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 8: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:31.907486 5065 raft_consensus.cc:686] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [term 9 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:32.040628 1568 leader_election.cc:302] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:32.037742 1598 raft_consensus.cc:2454] T 2355fe5dfc824ec78c044695be9d32de P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:32.038918 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3583c46ad2f4aa8875c06ef46cde18a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:32.041765 1599 raft_consensus.cc:3037] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:32.041916 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1355c134ca5f45b381fb2b12b1865adb" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:32.039973 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:32.045755 1448 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:32.049151 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:32.052317 2208 raft_consensus.cc:2379] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 9: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:32.053858 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:32.054917 1598 raft_consensus.cc:2379] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 6: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
I20260808 01:11:32.056236 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: true dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:32.059000 2171 leader_election.cc:302] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:32.022658 5185 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:11:32.059937 2171 leader_election.cc:302] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:32.064845 2171 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:32.024593 5216 consensus_meta.cc:287] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.565s user 0.004s sys 0.012s
I20260808 01:11:32.066156 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb799d94774eb18fed4dd8fdb5ac63" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:32.066593 1598 raft_consensus.cc:2454] T 55fb799d94774eb18fed4dd8fdb5ac63 P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 10.
I20260808 01:11:32.067171 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:32.069432 2171 leader_election.cc:302] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:32.070137 1598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:32.027092 5218 raft_consensus.cc:2781] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:11:32.027115 5120 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:32.073529 2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:32.065615 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0a7f6aa072473098cb2c51da278a83" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:32.047694 5099 raft_consensus.cc:2781] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:32.040738 5172 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:32.078752 1599 raft_consensus.cc:2454] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 9.
W20260808 01:11:32.082221 5097 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.466s user 0.015s sys 0.016s
I20260808 01:11:32.060322 5037 raft_consensus.cc:2781] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:11:32.062594 5275 raft_consensus.cc:2781] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:11:32.141963 5275 raft_consensus.cc:482] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:32.071620 5174 raft_consensus.cc:2726] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260808 01:11:32.143146 5213 consensus_meta.cc:287] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.536s user 0.011s sys 0.005s
I20260808 01:11:32.068980 5127 raft_consensus.cc:2781] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:32.143815 5127 raft_consensus.cc:482] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6b09dbefd24b4d52b385c4dff5697dc5)
I20260808 01:11:32.143600 5213 raft_consensus.cc:504] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:32.144289 5127 raft_consensus.cc:3037] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:32.144621 5241 leader_election.cc:288] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:32.079900 2200 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 10.
I20260808 01:11:32.034586 5090 consensus_queue.cc:1046] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.045s
I20260808 01:11:32.063953 5185 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:32.132359 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:32.079411 5181 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:32.102428 1489 leader_election.cc:302] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:32.136387 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:32.155439 2208 raft_consensus.cc:2454] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:32.136783 1445 raft_consensus.cc:1226] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260808 01:11:32.063982 5286 raft_consensus.cc:2781] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:32.138801 5204 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:32.161163 1408 leader_election.cc:302] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:32.139423 5254 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:32.068975 5015 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:32.140864 5065 consensus_queue.cc:237] T a668c2efd2dd4f5b946b70904c143c17 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.142369 5275 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:11:32.077134 5218 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:32.071769 5216 raft_consensus.cc:504] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.135159 5037 raft_consensus.cc:482] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:32.047106 5285 raft_consensus.cc:2781] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:32.153108 1439 raft_consensus.cc:1226] T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 LEADER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:32.027799 5053 raft_consensus.cc:504] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } attrs { replace: true } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.157280 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 1099) took 1025 ms. Trace:
I20260808 01:11:32.161176 5185 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:11:32.162501 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 60187.3us
I20260808 01:11:32.133067 5099 raft_consensus.cc:482] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:32.084776 5116 leader_election.cc:288] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:32.083784 5120 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:32.168478 5087 leader_election.cc:288] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:32.201876 2209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 1075) took 3231 ms. Trace:
I20260808 01:11:32.202229 2209 rpcz_store.cc:276] 0808 01:11:28.970584 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:29.074513 (+103929us) service_pool.cc:227] Handling call
0808 01:11:29.074875 (+ 362us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:29.272311 (+197436us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:32.201581 (+2929270us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b540004c754. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:32.201818 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":452018944}
I20260808 01:11:32.203341 2209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:32.168339 5171 consensus_queue.cc:1046] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:11:32.205029 5120 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.173086 5218 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:11:32.186537 5179 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:32.172336 4733 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:32.084131 5233 raft_consensus.cc:2726] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:11:32.071547 5258 raft_consensus.cc:2726] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:32.177376 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd5468b2c6e4509ae17a98f9509ff03" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:32.132205 5097 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.525s user 0.016s sys 0.017s
I20260808 01:11:32.178689 1444 raft_consensus.cc:2454] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:32.177160 2208 rpcz_store.cc:276] 0808 01:11:31.131870 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.156281 (+1024411us) service_pool.cc:227] Handling call
0808 01:11:32.156656 (+ 375us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:32.156904 (+ 248us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:32.156942 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:32.156992 (+ 50us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:32.157032 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:32.157098 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:32.157217 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:32.216923 1599 raft_consensus.cc:2379] T efd5468b2c6e4509ae17a98f9509ff03 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 8: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:32.182461 5053 leader_election.cc:288] T c12c84f631ee4ca7b405e28317db1b8a P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:32.163770 5204 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:32.185227 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 1097) took 1060 ms. Trace:
I20260808 01:11:32.219918 2204 rpcz_store.cc:276] 0808 01:11:31.125015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.817663 (+692648us) service_pool.cc:227] Handling call
0808 01:11:31.818066 (+ 403us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:31.871269 (+ 53203us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:32.185007 (+313738us) spinlock_profiling.cc:243] Waited 28.6 ms on lock 0x7b540059fbd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:32.185174 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":65889024}
I20260808 01:11:32.161592 5286 raft_consensus.cc:482] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:32.165882 5139 raft_consensus.cc:482] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:32.166170 4922 raft_consensus.cc:2781] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:32.198407 5285 raft_consensus.cc:686] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:32.195303 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1191) took 2584 ms. Trace:
I20260808 01:11:32.165472 5263 raft_consensus.cc:2781] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:11:32.168408 5189 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:32.206918 5094 raft_consensus.cc:504] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.178624 5037 raft_consensus.cc:3037] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:11:32.170439 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e354fdfbf2264912b39cd20be2cb19f8" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:32.136521 5172 raft_consensus.cc:504] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.248589 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:32.195011 5099 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:32.218956 2171 leader_election.cc:302] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:32.243619 2172 leader_election.cc:302] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:32.223657 2208 raft_consensus.cc:1226] T 6b1ad14dc361495f8e43a49a35526563 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:32.225827 2206 rpcz_store.cc:276] 0808 01:11:29.617996 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:29.743153 (+125157us) service_pool.cc:227] Handling call
0808 01:11:29.769944 (+ 26791us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:30.002437 (+232493us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:30.002473 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:30.002512 (+ 39us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:30.002557 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:32.194968 (+2192411us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b5400452754. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:32.195020 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:32.195227 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":534174592}
I20260808 01:11:32.201297 5213 leader_election.cc:288] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:32.217082 4887 raft_consensus.cc:482] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:32.231246 5019 consensus_queue.cc:1035] T 1ccc57605ae24bf4bacee4e4741bbe95 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.004s
I20260808 01:11:32.232476 5252 raft_consensus.cc:504] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.237113 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:32.227610 4922 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:32.215616 5179 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:32.232028 5139 raft_consensus.cc:504] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
W20260808 01:11:32.209370 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 275664us
I20260808 01:11:32.176788 5174 consensus_queue.cc:1046] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260808 01:11:32.227854 5263 raft_consensus.cc:482] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:32.292127 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:32.309875 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:32.225942 5286 raft_consensus.cc:3037] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:32.407490 1448 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:32.411669 5263 raft_consensus.cc:3037] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:11:32.414067 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 195175us
W20260808 01:11:32.414273 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 170745us
I20260808 01:11:32.423108 2201 raft_consensus.cc:2354] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 5. Current term is 6.
I20260808 01:11:32.411101 1599 raft_consensus.cc:3037] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:11:32.313138 5295 log.cc:924] Time spent T e354fdfbf2264912b39cd20be2cb19f8 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20260808 01:11:32.415498 5233 raft_consensus.cc:2726] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:11:32.416129 5097 raft_consensus.cc:2726] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:32.257416 4596 raft_consensus.cc:482] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:32.416841 1439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55042 (request call id 2533) took 1014 ms. Trace:
I20260808 01:11:32.442763 1439 rpcz_store.cc:276] 0808 01:11:31.402900 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.064854 (+661954us) service_pool.cc:227] Handling call
0808 01:11:32.065133 (+ 279us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:32.177760 (+112627us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:32.416604 (+238844us) spinlock_profiling.cc:243] Waited 87.7 ms on lock 0x7b540026c5d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:32.416781 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":201649024}
I20260808 01:11:32.416131 5273 raft_consensus.cc:3032] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:11:32.419338 1490 leader_election.cc:302] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:32.444216 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18b5317cac4512ad16a64664fab357" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:32.251855 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:32.420097 1568 leader_election.cc:302] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:32.445458 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:32.356549 5120 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:32.445981 1439 raft_consensus.cc:3037] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:32.368777 5292 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:32.449522 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:32.450290 1443 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:32.420586 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:32.410207 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49260 (request call id 836) took 1023 ms. Trace:
I20260808 01:11:32.451653 2208 rpcz_store.cc:276] 0808 01:11:31.387161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.223097 (+835936us) service_pool.cc:227] Handling call
0808 01:11:32.223424 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:32.409939 (+186515us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:32.410152 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:32.452132 1490 leader_election.cc:302] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:32.425885 2206 raft_consensus.cc:1226] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 11 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20260808 01:11:32.427644 4960 raft_consensus.cc:3037] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:32.429455 5213 raft_consensus.cc:482] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:32.431676 5252 leader_election.cc:288] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:32.432518 1516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2238) took 2816 ms. Trace:
I20260808 01:11:32.464825 1516 rpcz_store.cc:276] 0808 01:11:29.616527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:29.653003 (+ 36476us) service_pool.cc:227] Handling call
0808 01:11:29.675885 (+ 22882us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:30.248921 (+573036us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:32.432180 (+2183259us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x7b54000a73d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:32.432394 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1105060224}
I20260808 01:11:32.458520 4953 raft_consensus.cc:2781] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:11:32.413087 5204 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.466403 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:32.467046 1516 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:32.467880 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:32.468284 1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2713) took 1119 ms. Trace:
I20260808 01:11:32.468490 1522 raft_consensus.cc:3037] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:32.468650 1597 rpcz_store.cc:276] 0808 01:11:31.349187 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.790870 (+441683us) service_pool.cc:227] Handling call
0808 01:11:32.468119 (+677249us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400304b54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:32.468217 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":320208128}
I20260808 01:11:32.469213 5204 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:32.462742 5213 raft_consensus.cc:504] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.469905 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:32.470534 1597 raft_consensus.cc:2454] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 7.
I20260808 01:11:32.470963 1597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2715) took 1121 ms. Trace:
I20260808 01:11:32.471223 1597 rpcz_store.cc:276] 0808 01:11:31.349756 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.469559 (+1119803us) service_pool.cc:227] Handling call
0808 01:11:32.470906 (+ 1347us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:32.434582 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:32.342316 5216 leader_election.cc:288] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:32.348498 5172 leader_election.cc:288] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:32.439572 1518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2299) took 3218 ms. Trace:
I20260808 01:11:32.439821 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846d38a10c5846789f0b926b61905c0b" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:32.411893 4922 raft_consensus.cc:3037] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:32.452950 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:32.501894 2208 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 6.
I20260808 01:11:32.503158 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:32.412292 4887 raft_consensus.cc:504] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:32.462664 5047 raft_consensus.cc:1050] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d: attempting to promote NON_VOTER 95b661b03f1d460b9657e61d4b0247cd to VOTER
I20260808 01:11:32.451390 5273 raft_consensus.cc:729] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:32.466215 4953 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:32.286859 5094 leader_election.cc:288] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:32.472879 1439 raft_consensus.cc:2454] T d9e2d40d6b2943d883579bc4571ed4c1 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:32.335309 5218 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.500633 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:32.480022 1599 raft_consensus.cc:2454] T 417bc7d1279d48e89c3599b26117aaf9 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:32.501643 1518 rpcz_store.cc:276] 0808 01:11:29.221093 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:29.230413 (+ 9320us) service_pool.cc:227] Handling call
0808 01:11:29.253062 (+ 22649us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:29.579946 (+326884us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:29.579992 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:29.580048 (+ 56us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:29.580093 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:32.439330 (+2859237us) spinlock_profiling.cc:243] Waited 327 ms on lock 0x7b54000a5fd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:32.439379 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:32.439515 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":751050112}
I20260808 01:11:32.523770 1516 raft_consensus.cc:2454] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:32.524619 1522 raft_consensus.cc:2454] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 8.
I20260808 01:11:32.524964 2171 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
I20260808 01:11:32.525560 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "260b61a7a7f84a32bda47e79a38ac469" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:32.526790 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:32.504769 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b8d42642b84d3f9a2adbbaf3fd5417" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:32.505854 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:32.478785 1490 leader_election.cc:302] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:32.458477 4871 raft_consensus.cc:2781] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:32.510733 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:32.528218 2208 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:32.512274 1520 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:32.481840 1601 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 7.
I20260808 01:11:32.512614 1443 raft_consensus.cc:2454] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:32.344651 5127 raft_consensus.cc:504] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.476181 5089 raft_consensus.cc:3037] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:32.519860 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:32.521852 1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 708) took 1146 ms. Trace:
I20260808 01:11:32.500275 5213 leader_election.cc:288] T 516db2422f0644aebfd1999735efdbb8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:32.537871 1439 raft_consensus.cc:2379] T bc8c41cadc374229baf248366012f202 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 8: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:32.542524 1565 leader_election.cc:302] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd, a4862d0938c84fc4bd840e4c17acae78; no voters:
I20260808 01:11:32.543104 2171 leader_election.cc:302] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:32.538259 1599 rpcz_store.cc:276] 0808 01:11:31.375115 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.218499 (+843384us) service_pool.cc:227] Handling call
0808 01:11:32.521775 (+303276us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:32.538455 2172 leader_election.cc:302] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:32.341756 5285 consensus_queue.cc:237] T 4e0a7f6aa072473098cb2c51da278a83 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.557494 2208 raft_consensus.cc:2454] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:32.558949 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:32.559943 1408 leader_election.cc:302] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters:
I20260808 01:11:32.521800 4953 raft_consensus.cc:3037] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:32.564853 1439 raft_consensus.cc:1261] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260808 01:11:32.535392 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 12 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:32.458571 4596 raft_consensus.cc:504] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.541785 5266 raft_consensus.cc:482] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:32.527311 1522 raft_consensus.cc:2454] T c12c84f631ee4ca7b405e28317db1b8a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 5.
W20260808 01:11:32.540838 1568 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:45628 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 3 similar messages]
I20260808 01:11:32.460974 5292 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:32.538580 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:32.529095 5110 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:32.567401 2209 raft_consensus.cc:1226] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:32.558944 5172 raft_consensus.cc:2781] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election won for term 6
W20260808 01:11:32.539495 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 60791us
I20260808 01:11:32.565963 4733 raft_consensus.cc:2781] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:11:32.536676 5015 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:32.473414 5120 raft_consensus.cc:482] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:32.544775 5168 raft_consensus.cc:2726] T 3f18b5317cac4512ad16a64664fab357 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:11:32.605398 1516 raft_consensus.cc:2454] T a3ce33a519294ae1bce66c8aca8cf601 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:32.568506 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:32.611498 1520 raft_consensus.cc:2454] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 8.
I20260808 01:11:32.612185 2208 raft_consensus.cc:1261] T b3583c46ad2f4aa8875c06ef46cde18a P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Refusing update from remote peer 6b09dbefd24b4d52b385c4dff5697dc5: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260808 01:11:32.609828 1410 leader_election.cc:334] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 1.836s (SENT)
W20260808 01:11:32.607379 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 48181.7us
I20260808 01:11:32.640866 1522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:32.710125 1490 leader_election.cc:302] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:32.710484 1522 raft_consensus.cc:3037] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:32.711099 1410 leader_election.cc:302] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d; no voters: 6b09dbefd24b4d52b385c4dff5697dc5, a4862d0938c84fc4bd840e4c17acae78
W20260808 01:11:32.714116 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 104339us
I20260808 01:11:32.714726 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c1759594d4722abadf576352f9a98" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:32.715363 1520 raft_consensus.cc:2454] T a19c1759594d4722abadf576352f9a98 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 4.
I20260808 01:11:32.600217 5252 raft_consensus.cc:2781] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:32.570590 5213 raft_consensus.cc:2781] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:11:32.716034 1410 leader_election.cc:302] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:32.723917 2208 raft_consensus.cc:1226] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7. Ops: []
W20260808 01:11:32.694384 5282 consensus_meta.cc:287] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.740s user 0.010s sys 0.005s
I20260808 01:11:32.726565 1445 raft_consensus.cc:1226] T 260b61a7a7f84a32bda47e79a38ac469 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:32.729261 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:32.616353 5219 raft_consensus.cc:2781] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:11:32.731667 1518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:32.732421 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bc87d4fdb1436f875364cc78ec3fca" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:32.708967 5139 leader_election.cc:288] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:32.737594 1443 raft_consensus.cc:1226] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Rejecting Update request from peer 95b661b03f1d460b9657e61d4b0247cd for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260808 01:11:32.735474 1518 raft_consensus.cc:3037] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:32.723389 5019 raft_consensus.cc:2781] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:32.718097 5020 raft_consensus.cc:2726] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:11:32.719965 5172 raft_consensus.cc:686] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:32.717368 4871 raft_consensus.cc:686] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Becoming Leader. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
W20260808 01:11:32.752502 5302 log.cc:924] Time spent T 417bc7d1279d48e89c3599b26117aaf9 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20260808 01:11:32.753073 5218 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 11 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:32.756934 5172 consensus_queue.cc:237] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.728646 4733 raft_consensus.cc:686] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [term 8 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:32.729681 5168 raft_consensus.cc:2781] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:32.556507 1567 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:45638 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2726)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2728)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2430)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2719)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2725)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 714)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 701)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2721)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 718)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 715)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2722)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2723)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2730)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2441)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2734)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2444)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2724)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2735)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 720)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2438)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2435)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2736)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2731)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2732)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2437)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2439)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2440)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 719)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2729)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2442)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2434)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2733)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2433)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2436)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2424)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2702)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2703)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2425)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 691)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2426)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2431)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 700)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 702)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 705)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 712)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 716)
I20260808 01:11:32.604893 5185 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.646314 5110 raft_consensus.cc:504] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:32.748437 1522 raft_consensus.cc:2454] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 6.
I20260808 01:11:32.729048 5127 leader_election.cc:288] T 2355fe5dfc824ec78c044695be9d32de P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:32.793691 5303 consensus_queue.cc:1046] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:11:32.796516 5168 raft_consensus.cc:686] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Becoming Leader. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:32.613256 5266 raft_consensus.cc:504] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.798230 5168 consensus_queue.cc:237] T d9e2d40d6b2943d883579bc4571ed4c1 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:32.804927 4871 consensus_queue.cc:237] T ea62d4057ca44da4b68e6710052dfbe5 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:32.808364 5292 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:32.820595 5275 consensus_meta.cc:287] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.634s user 0.014s sys 0.001s
I20260808 01:11:32.821275 5275 raft_consensus.cc:504] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:32.823276 1599 raft_consensus.cc:2379] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 11: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:32.824970 1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2429) took 1669 ms. Trace:
I20260808 01:11:32.825380 1599 rpcz_store.cc:276] 0808 01:11:31.155517 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.558650 (+1403133us) service_pool.cc:227] Handling call
0808 01:11:32.824735 (+266085us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b54002094d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:32.824880 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":605864832}
I20260808 01:11:32.826638 1599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b19a284f2b41fe843e33e399ea8ea7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:32.827436 1604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2421) took 2491 ms (client timeout 1602 ms). Trace:
I20260808 01:11:32.828049 5228 raft_consensus.cc:482] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:11:32.828016 1604 rpcz_store.cc:269] 0808 01:11:30.336131 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:30.337930 (+ 1799us) service_pool.cc:227] Handling call
0808 01:11:32.827207 (+2489277us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540024f9d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:32.827352 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2639549824}
I20260808 01:11:32.828707 5228 raft_consensus.cc:3037] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:32.830505 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc54c3f2f5e04f63b88243c0d2d6babe" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:32.831317 1523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36560 (request call id 715) took 2450 ms. Trace:
W20260808 01:11:32.831512 1440 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55042 (request call id 2534) took 1389 ms (client timeout 1687 ms). Trace:
I20260808 01:11:32.831804 1523 rpcz_store.cc:276] 0808 01:11:30.381144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:30.411820 (+ 30676us) service_pool.cc:227] Handling call
0808 01:11:30.433677 (+ 21857us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:30.479439 (+ 45762us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:32.830986 (+2351547us) spinlock_profiling.cc:243] Waited 44.9 ms on lock 0x7b540005d054. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:32.831221 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":103168768}
W20260808 01:11:32.832433 1440 rpcz_store.cc:269] 0808 01:11:31.442670 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.830167 (+1387497us) service_pool.cc:227] Handling call
0808 01:11:32.831447 (+ 1280us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:32.833616 1523 raft_consensus.cc:1226] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:32.834390 1523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2331) took 1227 ms. Trace:
I20260808 01:11:32.732858 5213 raft_consensus.cc:686] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:32.739584 5219 raft_consensus.cc:482] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:11:32.733227 5252 raft_consensus.cc:686] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:32.834765 1523 rpcz_store.cc:276] 0808 01:11:31.607269 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.833011 (+1225742us) service_pool.cc:227] Handling call
0808 01:11:32.833401 (+ 390us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:32.834058 (+ 657us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:32.834261 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:32.862089 2172 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 155848us
I20260808 01:11:32.864348 5311 consensus_queue.cc:1046] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:11:32.735548 5282 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.741496 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:32.717115 5120 raft_consensus.cc:504] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:32.716144 5015 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.600600 5273 consensus_queue.cc:260] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.743687 5019 raft_consensus.cc:482] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:32.784722 1448 raft_consensus.cc:2379] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 12: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:32.876348 1522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2329) took 1289 ms. Trace:
W20260808 01:11:32.788967 5037 consensus_meta.cc:287] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.544s user 0.007s sys 0.006s
I20260808 01:11:32.790504 4596 leader_election.cc:288] T 31f1af2e7c504c919e34f12663452f2a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:32.792634 4733 consensus_queue.cc:237] T 921f3eb5150b4105a4b68355cca8cb7a P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.795461 5303 raft_consensus.cc:3032] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:11:32.829463 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:32.869051 4833 leader_election.cc:288] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:32.871668 5252 consensus_queue.cc:237] T 260b61a7a7f84a32bda47e79a38ac469 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.872931 5179 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:32.765347 1516 raft_consensus.cc:1226] T 846d38a10c5846789f0b926b61905c0b P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260808 01:11:32.855087 5139 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:32.897627 1516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2327) took 1335 ms. Trace:
I20260808 01:11:32.898011 1516 rpcz_store.cc:276] 0808 01:11:31.562481 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.737477 (+1174996us) service_pool.cc:227] Handling call
0808 01:11:32.761144 (+ 23667us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:32.897340 (+136196us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:32.897569 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:32.856392 5185 leader_election.cc:288] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 12 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:32.878686 1520 raft_consensus.cc:3037] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Advancing to term 6
I20260808 01:11:32.886108 5273 raft_consensus.cc:3037] T efd5468b2c6e4509ae17a98f9509ff03 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:32.881460 5223 raft_consensus.cc:482] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:32.888553 5120 leader_election.cc:288] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:32.852774 5219 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:32.890276 1518 raft_consensus.cc:2454] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
W20260808 01:11:32.901518 1490 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.65:38291, user_credentials={real_user=slave}} blocked reactor thread for 191256us
I20260808 01:11:32.883739 1522 rpcz_store.cc:276] 0808 01:11:31.587263 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.773148 (+1185885us) service_pool.cc:227] Handling call
0808 01:11:32.773540 (+ 392us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:32.796691 (+ 23151us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:32.796722 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:32.797206 (+ 484us) log.cc:842] Serialized 26 byte log entry
0808 01:11:32.858625 (+ 61419us) raft_consensus.cc:1631] Marking committed up to 2
0808 01:11:32.858668 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:32.858741 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:32.876130 (+ 17389us) raft_consensus.cc:1671] finished
0808 01:11:32.876147 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:32.876288 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22703,"raft.queue_time_us":24204,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"thread_start_us":60941,"threads_started":1,"wal-append.queue_time_us":61501}
I20260808 01:11:32.899956 1440 raft_consensus.cc:1261] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260808 01:11:32.890539 5019 raft_consensus.cc:3037] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:32.882174 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:32.903808 1518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2333) took 1283 ms. Trace:
I20260808 01:11:32.902460 1410 leader_election.cc:302] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:32.904155 1523 raft_consensus.cc:2396] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260808 01:11:32.902513 1490 leader_election.cc:302] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:32.905481 5179 raft_consensus.cc:2726] T d6b8d42642b84d3f9a2adbbaf3fd5417 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:11:32.905653 1523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36560 (request call id 731) took 1076 ms. Trace:
I20260808 01:11:32.906042 1523 rpcz_store.cc:276] 0808 01:11:31.829636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.904856 (+1075220us) service_pool.cc:227] Handling call
0808 01:11:32.905217 (+ 361us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:32.905384 (+ 167us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:32.905405 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:32.905446 (+ 41us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:32.905472 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:32.905516 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:32.905606 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:32.907197 5179 consensus_queue.cc:1046] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:11:32.908339 5279 raft_consensus.cc:3032] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:32.908627 5279 raft_consensus.cc:729] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:32.908659 1490 leader_election.cc:302] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:32.909271 5279 consensus_queue.cc:260] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.909673 5179 raft_consensus.cc:2726] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:32.910431 5279 raft_consensus.cc:3037] T 260b61a7a7f84a32bda47e79a38ac469 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:32.904688 5185 raft_consensus.cc:2781] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:32.911621 5185 raft_consensus.cc:686] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:32.911806 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:32.912606 5185 consensus_queue.cc:237] T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:32.913005 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:32.914896 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:32.915510 1445 raft_consensus.cc:2454] T 58c8c7709e68463182371aa8224d2da2 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 7.
I20260808 01:11:32.904217 1518 rpcz_store.cc:276] 0808 01:11:31.620105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.903019 (+1282914us) service_pool.cc:227] Handling call
0808 01:11:32.903323 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:32.903504 (+ 181us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:32.903536 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:32.903576 (+ 40us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:32.903609 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:32.903661 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:32.903755 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:32.917796 1490 leader_election.cc:302] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, a4862d0938c84fc4bd840e4c17acae78
I20260808 01:11:32.918893 5168 raft_consensus.cc:3037] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:32.924281 1522 raft_consensus.cc:1261] T f6bb0551ce74437284392981d2921446 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 01:11:32.927073 5020 consensus_queue.cc:1035] T f6bb0551ce74437284392981d2921446 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:11:32.929925 1490 leader_election.cc:302] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5; no voters:
I20260808 01:11:32.891559 5303 raft_consensus.cc:729] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:32.890111 5000 consensus_queue.cc:1046] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20260808 01:11:32.957499 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:32.966521 5088 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:11:32.934396 5110 leader_election.cc:288] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:32.892179 5037 raft_consensus.cc:504] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:32.985811 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a668c2efd2dd4f5b946b70904c143c17" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:32.987804 2208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e59ebd6ca6845789ae50d410b699ea7" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:32.988587 2208 raft_consensus.cc:3037] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:11:32.903213 5292 consensus_queue.cc:1046] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.084s
I20260808 01:11:32.979607 5275 leader_election.cc:288] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 11 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:32.901687 5139 raft_consensus.cc:504] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:33.003865 2204 raft_consensus.cc:1226] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 9. Current term is 10. Ops: []
I20260808 01:11:32.978425 5015 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:33.010980 1567 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:45628 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2430)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2719)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2725)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 714)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 701)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2721)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 718)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 715)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2722)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2723)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2730)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2441)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2734)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2444)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2724)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2735)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 720)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2438)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2435)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2736)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2731)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2732)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2437)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2439)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2440)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2446)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 719)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2729)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2442)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2448)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2434)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2733)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2433)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2436)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2424)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2702)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2703)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2425)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 691)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2426)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2431)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 700)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 702)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 705)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 712)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 716)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2445)
I20260808 01:11:33.015082 2208 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 12.
I20260808 01:11:33.010975 1520 raft_consensus.cc:2454] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 6.
I20260808 01:11:33.016497 1408 leader_election.cc:302] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 95b661b03f1d460b9657e61d4b0247cd; no voters: 6b09dbefd24b4d52b385c4dff5697dc5
I20260808 01:11:32.975116 5312 raft_consensus.cc:2781] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:11:32.902693 5223 raft_consensus.cc:504] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:33.033074 1444 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55042 (request call id 2530) took 2478 ms (client timeout 1714 ms). Trace:
W20260808 01:11:33.033638 1444 rpcz_store.cc:269] 0808 01:11:30.554434 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:30.664836 (+110402us) service_pool.cc:227] Handling call
0808 01:11:33.032798 (+2367962us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54000e74d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:33.032945 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3479128832}
I20260808 01:11:33.036823 5219 raft_consensus.cc:504] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:33.038648 5219 leader_election.cc:288] T 36f2393bca80434dbdc089722b52edd1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:32.963909 5213 consensus_queue.cc:237] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:33.041841 2202 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1189) took 3442 ms (client timeout 2192 ms). Trace:
I20260808 01:11:33.024206 5312 raft_consensus.cc:482] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:11:33.042588 2202 rpcz_store.cc:269] 0808 01:11:29.609842 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:29.617155 (+ 7313us) service_pool.cc:227] Handling call
0808 01:11:33.041678 (+3424523us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b54005d0554. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:33.041782 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":296163072}
I20260808 01:11:33.027438 5185 raft_consensus.cc:3032] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 LEADER]: Stepping down as leader of term 7
W20260808 01:11:33.031060 5049 consensus_meta.cc:287] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 1.054s user 0.012s sys 0.007s
I20260808 01:11:33.045504 1597 raft_consensus.cc:2454] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 5.
I20260808 01:11:33.029971 5292 raft_consensus.cc:2781] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:11:33.047761 1600 raft_consensus.cc:595] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Received request to transfer leadership to TS 95b661b03f1d460b9657e61d4b0247cd
I20260808 01:11:33.049340 1600 raft_consensus.cc:603] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:11:33.048069 2202 raft_consensus.cc:1261] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:11:33.045684 5312 raft_consensus.cc:3037] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:33.050979 2202 raft_consensus.cc:1226] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:33.031437 5089 raft_consensus.cc:2726] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260808 01:11:33.044308 4922 consensus_meta.cc:287] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d: Time spent flushing consensus metadata: real 0.538s user 0.009s sys 0.005s
I20260808 01:11:33.052397 2208 raft_consensus.cc:1226] T bc54c3f2f5e04f63b88243c0d2d6babe P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:33.055300 2202 raft_consensus.cc:1226] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 7. Current term is 8. Ops: []
W20260808 01:11:33.052367 5248 consensus_meta.cc:287] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 1.061s user 0.004s sys 0.015s
I20260808 01:11:33.047087 5185 raft_consensus.cc:729] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:33.008006 5303 consensus_queue.cc:260] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:33.067914 1599 raft_consensus.cc:2454] T 93b19a284f2b41fe843e33e399ea8ea7 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 6.
I20260808 01:11:33.054081 5282 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 11 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:33.056633 5292 raft_consensus.cc:686] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 12 LEADER]: Becoming Leader. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:33.093107 5292 consensus_queue.cc:237] T 1e59ebd6ca6845789ae50d410b699ea7 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:33.105762 5089 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:33.106727 5089 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:33.108328 1602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2406) took 4968 ms (client timeout 1836 ms). Trace:
W20260808 01:11:33.108800 1602 rpcz_store.cc:269] 0808 01:11:28.139973 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:28.144524 (+ 4551us) service_pool.cc:227] Handling call
0808 01:11:33.108091 (+4963567us) spinlock_profiling.cc:243] Waited 1.9 s on lock 0x7b54001e9954. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:33.108209 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4370463616}
I20260808 01:11:33.108601 5185 consensus_queue.cc:260] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:33.109930 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.110718 1602 raft_consensus.cc:2379] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 6: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:33.110867 5139 leader_election.cc:288] T 5353a0d85aff48078011ca6efaf9ddfa P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:33.110615 5089 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:33.112326 5316 log.cc:924] Time spent T 9ea0da159669492a90fc257d91f4ba5c P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.053s user 0.000s sys 0.002s
I20260808 01:11:33.116397 1525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2326) took 1930 ms. Trace:
I20260808 01:11:33.117024 1525 rpcz_store.cc:276] 0808 01:11:31.186059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.187263 (+ 1204us) service_pool.cc:227] Handling call
0808 01:11:31.202759 (+ 15496us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:31.216899 (+ 14140us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:33.116130 (+1899231us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b540012eb54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:33.116333 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9400192}
I20260808 01:11:33.059320 4922 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:33.085291 5303 raft_consensus.cc:3037] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:11:33.111186 1602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2430) took 1931 ms (client timeout 2556 ms). Trace:
W20260808 01:11:33.126183 1602 rpcz_store.cc:269] 0808 01:11:31.179730 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.109658 (+1929928us) service_pool.cc:227] Handling call
0808 01:11:33.111131 (+ 1473us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:33.129348 4922 leader_election.cc:288] T 58547863eabc48eda1552ed4240f68d3 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:33.131213 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 587205us
I20260808 01:11:33.132119 5185 raft_consensus.cc:3037] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:33.138370 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.146631 1602 raft_consensus.cc:2354] T 498a0710a54e491dbd1a79ce64033f62 P a4862d0938c84fc4bd840e4c17acae78 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 10. Current term is 11.
I20260808 01:11:33.147147 1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2719) took 1560 ms. Trace:
I20260808 01:11:33.150691 5165 consensus_queue.cc:1035] T b3583c46ad2f4aa8875c06ef46cde18a P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:11:33.152179 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:33.155049 5258 raft_consensus.cc:3032] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:11:33.157990 5258 raft_consensus.cc:729] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:33.158321 5168 raft_consensus.cc:2726] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:33.159456 5258 consensus_queue.cc:260] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:33.161265 5258 raft_consensus.cc:3037] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:33.072293 5228 raft_consensus.cc:504] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:33.102598 5037 leader_election.cc:288] T 44f40a08ace944a88ad5bf8ba22f2a40 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 10 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:33.157403 5031 leader_election.cc:288] T 5a8d1d748c8f4d44942ad2f2e57e9162 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
W20260808 01:11:33.194831 1438 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55040 (request call id 2348) took 2801 ms (client timeout 1784 ms). Trace:
W20260808 01:11:33.165794 2171 leader_election.cc:334] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 1.827s (SENT)
I20260808 01:11:33.155287 1602 rpcz_store.cc:276] 0808 01:11:31.586876 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.138103 (+1551227us) service_pool.cc:227] Handling call
0808 01:11:33.147095 (+ 8992us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:33.198113 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a539e94f6f744f20b65ea3fa4bbf8be4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.199141 1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2725) took 1326 ms. Trace:
W20260808 01:11:33.133616 5099 consensus_meta.cc:287] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.716s user 0.010s sys 0.005s
W20260808 01:11:33.156634 1594 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2412) took 3143 ms (client timeout 1805 ms). Trace:
I20260808 01:11:33.180733 5228 leader_election.cc:288] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:33.197101 2205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 1087) took 3084 ms. Trace:
I20260808 01:11:33.199513 1602 rpcz_store.cc:276] 0808 01:11:31.872471 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.197771 (+1325300us) service_pool.cc:227] Handling call
0808 01:11:33.199075 (+ 1304us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:33.195247 1438 rpcz_store.cc:269] 0808 01:11:30.393758 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:30.405507 (+ 11749us) service_pool.cc:227] Handling call
0808 01:11:33.194649 (+2789142us) spinlock_profiling.cc:243] Waited 92.1 ms on lock 0x7b54001d2c54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:33.194757 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":211784064}
I20260808 01:11:33.223181 5099 raft_consensus.cc:504] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:33.337443 2204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 1104) took 1800 ms. Trace:
I20260808 01:11:33.385515 2204 rpcz_store.cc:276] 0808 01:11:31.536576 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.224197 (+687621us) service_pool.cc:227] Handling call
0808 01:11:32.224550 (+ 353us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:33.014586 (+790036us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:33.337087 (+322501us) spinlock_profiling.cc:243] Waited 779 ms on lock 0x7b54003656d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:33.337350 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1791776640}
W20260808 01:11:33.384593 1442 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60392 (request call id 886) took 3282 ms (client timeout 2318 ms). Trace:
W20260808 01:11:33.340193 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 135763us
I20260808 01:11:33.340354 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "212d8f7da4b84fae81802fe11cdd4b81" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:33.387542 1602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 714) took 1701 ms (client timeout 2141 ms). Trace:
W20260808 01:11:33.388182 1602 rpcz_store.cc:269] 0808 01:11:31.685783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.232429 (+1546646us) service_pool.cc:227] Handling call
0808 01:11:33.387481 (+155052us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:33.387409 2171 leader_election.cc:302] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
W20260808 01:11:33.211637 1594 rpcz_store.cc:269] 0808 01:11:30.013763 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:30.052078 (+ 38315us) service_pool.cc:227] Handling call
0808 01:11:33.156439 (+3104361us) spinlock_profiling.cc:243] Waited 943 ms on lock 0x7b540028f5d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:33.156566 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2169800576}
I20260808 01:11:33.389412 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921f3eb5150b4105a4b68355cca8cb7a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:33.194105 5165 consensus_queue.cc:1046] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.087s
I20260808 01:11:33.223541 2205 rpcz_store.cc:276] 0808 01:11:30.112961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:30.113450 (+ 489us) service_pool.cc:227] Handling call
0808 01:11:30.113734 (+ 284us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:30.248068 (+134334us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:33.190585 (+2942517us) spinlock_profiling.cc:243] Waited 80.8 ms on lock 0x7b5400494154. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:33.197016 (+ 6431us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":185773440}
I20260808 01:11:33.403047 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:33.403476 2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:33.404124 2205 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:11:33.188984 5053 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.028s sys 0.028s
W20260808 01:11:33.344178 1601 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2710) took 2969 ms (client timeout 1950 ms). Trace:
W20260808 01:11:33.410728 1601 rpcz_store.cc:269] 0808 01:11:30.375148 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:30.673078 (+297930us) service_pool.cc:227] Handling call
0808 01:11:33.338651 (+2665573us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54002dd2d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:33.344108 (+ 5457us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3366602496}
I20260808 01:11:33.197871 5103 consensus_queue.cc:1046] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.110s
W20260808 01:11:33.356731 1597 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 707) took 1982 ms (client timeout 2227 ms). Trace:
W20260808 01:11:33.411895 1597 rpcz_store.cc:269] 0808 01:11:31.374759 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.471817 (+1097058us) service_pool.cc:227] Handling call
0808 01:11:33.356521 (+884704us) spinlock_profiling.cc:243] Waited 518 ms on lock 0x7b5400055ad4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:33.356653 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1190493440}
I20260808 01:11:33.345571 1521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2324) took 2167 ms. Trace:
I20260808 01:11:33.413007 1521 rpcz_store.cc:276] 0808 01:11:31.178008 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.178148 (+ 140us) service_pool.cc:227] Handling call
0808 01:11:31.178465 (+ 317us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:31.242021 (+ 63556us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:31.242068 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:31.242126 (+ 58us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:31.242171 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:33.336247 (+2094076us) spinlock_profiling.cc:243] Waited 63.3 ms on lock 0x7b54000c7454. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:33.345337 (+ 9090us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:33.345501 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":145596672}
I20260808 01:11:33.414805 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:33.365805 1517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2304) took 4026 ms. Trace:
W20260808 01:11:33.386590 1442 rpcz_store.cc:269] 0808 01:11:30.101929 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.384389 (+3282460us) service_pool.cc:214] Skipping call since client already timed out
0808 01:11:33.384522 (+ 133us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 01:11:33.415627 1517 rpcz_store.cc:276] 0808 01:11:29.339153 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:29.415868 (+ 76715us) service_pool.cc:227] Handling call
0808 01:11:30.545736 (+1129868us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:30.659006 (+113270us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:33.365535 (+2706529us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540023d7d0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:33.365726 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2597020416}
W20260808 01:11:33.387431 1446 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55040 (request call id 2352) took 2017 ms (client timeout 1531 ms). Trace:
I20260808 01:11:33.194505 5089 raft_consensus.cc:482] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:33.385213 1519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2322) took 2868 ms. Trace:
I20260808 01:11:33.390321 1594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 718) took 1217 ms. Trace:
I20260808 01:11:33.188841 5168 raft_consensus.cc:482] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:33.390991 5263 consensus_meta.cc:287] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.950s user 0.005s sys 0.012s
I20260808 01:11:33.408923 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.409304 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:33.417583 1594 rpcz_store.cc:276] 0808 01:11:32.173351 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.389122 (+1215771us) service_pool.cc:227] Handling call
0808 01:11:33.390258 (+ 1136us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:33.433764 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 46473.5us
I20260808 01:11:33.433830 1604 raft_consensus.cc:2379] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 6: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:33.435230 2171 leader_election.cc:302] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d
W20260808 01:11:33.434160 1602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2721) took 1776 ms (client timeout 2222 ms). Trace:
W20260808 01:11:33.436581 1602 rpcz_store.cc:269] 0808 01:11:31.657755 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.408678 (+1750923us) service_pool.cc:227] Handling call
0808 01:11:33.434098 (+ 25420us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:33.437114 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.437553 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a11329d12c4ba4abd2b4e0e29816c7" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:33.384584 5322 log.cc:924] Time spent T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5: Append to log took a long time: real 0.341s user 0.000s sys 0.006s
I20260808 01:11:33.438210 1602 raft_consensus.cc:2454] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 5.
I20260808 01:11:33.434158 2204 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:11:33.416980 1519 rpcz_store.cc:276] 0808 01:11:30.516733 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:30.516977 (+ 244us) service_pool.cc:227] Handling call
0808 01:11:30.517377 (+ 400us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:30.673977 (+156600us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:30.674008 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:30.674517 (+ 509us) log.cc:842] Serialized 26 byte log entry
0808 01:11:30.927444 (+252927us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:31.417422 (+489978us) log.cc:842] Serialized 14 byte log entry
0808 01:11:31.443093 (+ 25671us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:33.300920 (+1857827us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b540012d754. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:33.347516 (+ 46596us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:33.347583 (+ 67us) raft_consensus.cc:1671] finished
0808 01:11:33.384948 (+ 37365us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:33.385142 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10573,"prepare.queue_time_us":600321,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":29123,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":20104,"spinlock_wait_cycles":309533440,"thread_start_us":733744,"threads_started":2,"wal-append.queue_time_us":508246,"wal-append.run_cpu_time_us":2881,"wal-append.run_wall_time_us":1287642}
I20260808 01:11:33.447930 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
W20260808 01:11:33.416810 1446 rpcz_store.cc:269] 0808 01:11:31.328689 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.329606 (+ 917us) service_pool.cc:227] Handling call
0808 01:11:33.300134 (+1970528us) spinlock_profiling.cc:243] Waited 618 ms on lock 0x7b540014cdd4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:33.346228 (+ 46094us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1420427008}
I20260808 01:11:33.450054 2203 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 1088) took 3318 ms. Trace:
I20260808 01:11:33.450366 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:33.450436 2203 rpcz_store.cc:276] 0808 01:11:30.131351 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:30.131553 (+ 202us) service_pool.cc:227] Handling call
0808 01:11:30.132028 (+ 475us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:31.637503 (+1505475us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:33.449772 (+1812269us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b540067ded4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:33.449989 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3460211456}
I20260808 01:11:33.451170 1520 raft_consensus.cc:2379] T 87907e719e4c46c4a19015f3f3bba308 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 6: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:33.451640 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
I20260808 01:11:33.422617 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1ad14dc361495f8e43a49a35526563" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.442601 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:33.443861 2171 leader_election.cc:302] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95b661b03f1d460b9657e61d4b0247cd; no voters: 3fb1a6380d27401a831cfe5f86a4cc9d, 6b09dbefd24b4d52b385c4dff5697dc5
W20260808 01:11:33.390249 5286 consensus_meta.cc:287] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.954s user 0.018s sys 0.000s
I20260808 01:11:33.454579 1525 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 5.
I20260808 01:11:33.446156 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.456000 2171 leader_election.cc:302] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b09dbefd24b4d52b385c4dff5697dc5, 95b661b03f1d460b9657e61d4b0247cd; no voters:
W20260808 01:11:33.393532 5235 log.cc:924] Time spent T 002e3bea322d445e8f5f8f09e2442db4 P 95b661b03f1d460b9657e61d4b0247cd: Append to log took a long time: real 0.171s user 0.009s sys 0.002s
I20260808 01:11:33.453797 1596 raft_consensus.cc:2454] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 7.
W20260808 01:11:33.474262 1596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 701) took 2121 ms (client timeout 2480 ms). Trace:
I20260808 01:11:33.455960 1594 raft_consensus.cc:2454] T 704a8e4165cc49079ada0c7993c21187 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 9.
W20260808 01:11:33.475625 1594 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2723) took 1618 ms (client timeout 2118 ms). Trace:
W20260808 01:11:33.476060 1594 rpcz_store.cc:269] 0808 01:11:31.857293 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.445859 (+1588566us) service_pool.cc:227] Handling call
0808 01:11:33.475554 (+ 29695us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:33.477202 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.472388 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:33.474787 1596 rpcz_store.cc:269] 0808 01:11:31.353068 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.387600 (+2034532us) service_pool.cc:227] Handling call
0808 01:11:33.474187 (+ 86587us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:33.481853 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0108cd20b4bb4957b8a59eebd68479e4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.485013 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002e3bea322d445e8f5f8f09e2442db4" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd" is_pre_election: true
W20260808 01:11:33.418488 5053 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.773s user 0.028s sys 0.028s
I20260808 01:11:33.487217 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0008d696f943a3aaa520463af63298" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:33.489606 1520 raft_consensus.cc:2396] T 4f0008d696f943a3aaa520463af63298 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260808 01:11:33.488392 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.495015 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80a0f28c7c94d45b7983fc0b1386caf" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.489542 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bc7d1279d48e89c3599b26117aaf9" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:33.498936 1525 raft_consensus.cc:2379] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 6: Already voted for candidate 95b661b03f1d460b9657e61d4b0247cd in this term.
I20260808 01:11:33.503966 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:33.504992 1517 raft_consensus.cc:2379] T 9ea0da159669492a90fc257d91f4ba5c P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 6: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:33.490834 1602 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 7.
I20260808 01:11:33.497447 1594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2724) took 1632 ms. Trace:
I20260808 01:11:33.506184 1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2441) took 1076 ms. Trace:
I20260808 01:11:33.507718 1602 rpcz_store.cc:276] 0808 01:11:32.430206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.473735 (+1043529us) service_pool.cc:227] Handling call
0808 01:11:33.506103 (+ 32368us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:33.505586 1597 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2722) took 1759 ms (client timeout 2209 ms). Trace:
W20260808 01:11:33.509500 1597 rpcz_store.cc:269] 0808 01:11:31.745698 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.437943 (+1692245us) service_pool.cc:227] Handling call
0808 01:11:33.505510 (+ 67567us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:33.507184 1594 rpcz_store.cc:276] 0808 01:11:31.864851 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.481488 (+1616637us) service_pool.cc:227] Handling call
0808 01:11:33.497340 (+ 15852us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:33.512200 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ce33a519294ae1bce66c8aca8cf601" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:33.514070 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 58199.1us
I20260808 01:11:33.506170 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:33.522779 1525 raft_consensus.cc:2454] T 498a0710a54e491dbd1a79ce64033f62 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 9.
I20260808 01:11:33.468513 5037 raft_consensus.cc:2781] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:11:33.528513 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516db2422f0644aebfd1999735efdbb8" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.539623 1596 raft_consensus.cc:2454] T 516db2422f0644aebfd1999735efdbb8 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 8.
I20260808 01:11:33.540243 1596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2444) took 1011 ms. Trace:
I20260808 01:11:33.540660 1596 rpcz_store.cc:276] 0808 01:11:32.529013 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.481056 (+952043us) service_pool.cc:227] Handling call
0808 01:11:33.540177 (+ 59121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:33.521438 1594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2438) took 1092 ms. Trace:
I20260808 01:11:33.536067 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:33.544175 1594 rpcz_store.cc:276] 0808 01:11:32.429265 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.511901 (+1082636us) service_pool.cc:227] Handling call
0808 01:11:33.521291 (+ 9390us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:33.546046 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.547093 1594 raft_consensus.cc:2379] T bcdecb8f50de443095f848f2500fd0a8 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 in current term 5: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
W20260808 01:11:33.549235 5277 log.cc:924] Time spent T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 2.041s user 0.005s sys 0.005s
I20260808 01:11:33.551101 1523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58266 (request call id 2336) took 1561 ms. Trace:
I20260808 01:11:33.552734 1447 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60392 (request call id 901) took 2477 ms. Trace:
I20260808 01:11:33.553187 1447 rpcz_store.cc:276] 0808 01:11:31.074994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.075180 (+ 186us) service_pool.cc:227] Handling call
0808 01:11:31.094884 (+ 19704us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:31.243027 (+148143us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:31.243069 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:31.248098 (+ 5029us) log.cc:842] Serialized 26 byte log entry
0808 01:11:31.408600 (+160502us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:31.408640 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:31.609664 (+201024us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b540030b6d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:31.609712 (+ 48us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:33.385868 (+1776156us) spinlock_profiling.cc:243] Waited 911 us on lock 0x7b44000232cc. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34995bf11 00007fe349a29a95 00007fe349a2a4b2 00007fe34c6d2511 00007fe34c6cde1a 00007fe34c6c1bee 00007fe34c6c0f36 00007fe34c6c0b58 00007fe34c6ca551 00007fe34c6ca509 00007fe34c6ca499
0808 01:11:33.552456 (+166588us) raft_consensus.cc:1671] finished
0808 01:11:33.552483 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:33.552648 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23859,"raft.queue_time_us":32462,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":14120,"spinlock_wait_cycles":273616384,"thread_start_us":134220,"threads_started":1,"wal-append.queue_time_us":211648}
I20260808 01:11:33.556679 2208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 1108) took 1269 ms. Trace:
I20260808 01:11:33.557120 2208 rpcz_store.cc:276] 0808 01:11:32.287735 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.015957 (+728222us) service_pool.cc:227] Handling call
0808 01:11:33.016266 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:33.059484 (+ 43218us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:33.556386 (+496902us) spinlock_profiling.cc:243] Waited 35.9 ms on lock 0x7b54004835d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:33.556593 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":82471680}
I20260808 01:11:33.551561 1523 rpcz_store.cc:276] 0808 01:11:31.989679 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.907704 (+918025us) service_pool.cc:227] Handling call
0808 01:11:32.908194 (+ 490us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:33.549854 (+641660us) spinlock_profiling.cc:243] Waited 3.07 ms on lock 0x7b54003ea420. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34c9d30d8 00007fe34ca3efed 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c
0808 01:11:33.549930 (+ 76us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:33.549967 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:33.550029 (+ 62us) raft_consensus.cc:1631] Marking committed up to 5
0808 01:11:33.550094 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:33.550806 (+ 712us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b540005d554. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:33.550867 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:33.551024 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1704,"raft.queue_time_us":4917,"spinlock_wait_cycles":342904832}
I20260808 01:11:33.564618 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9890c6b79a40fcbbf70e44142a95d0" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:33.567402 1594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2736) took 1058 ms. Trace:
I20260808 01:11:33.567976 1594 rpcz_store.cc:276] 0808 01:11:32.508761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.545726 (+1036965us) service_pool.cc:227] Handling call
0808 01:11:33.567318 (+ 21592us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:33.569149 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bb58b6f7c14c22814cfac6d2b2bf6d" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.411931 5019 raft_consensus.cc:504] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:33.574877 1523 raft_consensus.cc:2396] T ca9890c6b79a40fcbbf70e44142a95d0 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260808 01:11:33.519237 4953 consensus_meta.cc:287] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.916s user 0.012s sys 0.008s
I20260808 01:11:33.612335 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2355fe5dfc824ec78c044695be9d32de" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:33.612571 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea62d4057ca44da4b68e6710052dfbe5" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:33.613413 1525 raft_consensus.cc:3037] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Advancing to term 7
I20260808 01:11:33.614557 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:33.616346 4953 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:33.572172 5174 consensus_queue.cc:1035] T 4f0008d696f943a3aaa520463af63298 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.045s
I20260808 01:11:33.617755 1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:51424 (request call id 25) took 5296 ms. Trace:
I20260808 01:11:33.618218 1600 rpcz_store.cc:276] 0808 01:11:28.321578 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:28.750368 (+428790us) service_pool.cc:227] Handling call
0808 01:11:33.617506 (+4867138us) spinlock_profiling.cc:243] Waited 4.22 s on lock 0x7b5400055ad4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca39bc3 00007fe35198381a 00007fe34b076d04 00007fe34b076c92 00007fe34b076bc1 00007fe34b076b3c 00007fe34b075df2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:33.617636 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9709848320}
I20260808 01:11:33.435743 5089 raft_consensus.cc:504] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:33.578150 1597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7eb407c2934b7f9db6fa7a3ea0fae5" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:33.630927 1597 raft_consensus.cc:3037] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:33.579478 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.560714 5179 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:33.636889 5179 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:33.584100 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:33.644285 1596 raft_consensus.cc:3037] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:11:33.584909 2208 raft_consensus.cc:1226] T a83a963773094d6db6495b7009fd8690 P 95b661b03f1d460b9657e61d4b0247cd [term 9 LEADER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260808 01:11:33.590838 5220 raft_consensus.cc:2726] T 43bc87d4fdb1436f875364cc78ec3fca P 95b661b03f1d460b9657e61d4b0247cd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:11:33.562522 5037 raft_consensus.cc:482] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3fb1a6380d27401a831cfe5f86a4cc9d)
I20260808 01:11:33.608265 1331 maintenance_manager.cc:419] P 5a043bcc76cc4126b15e712346791f23: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033349
I20260808 01:11:33.609328 5227 raft_consensus.cc:2781] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:11:33.610965 5099 leader_election.cc:288] T d9e2d40d6b2943d883579bc4571ed4c1 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093)
I20260808 01:11:33.571259 4922 raft_consensus.cc:482] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:33.619571 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6bb0551ce74437284392981d2921446" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.570529 1594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2731) took 1065 ms. Trace:
I20260808 01:11:33.625269 1523 raft_consensus.cc:1226] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6. Ops: []
I20260808 01:11:33.538548 5074 raft_consensus.cc:2781] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:11:33.629277 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:33.726094 1516 raft_consensus.cc:2396] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260808 01:11:33.442569 5263 raft_consensus.cc:504] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:33.472790 5286 raft_consensus.cc:504] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:33.593228 1604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2728) took 1489 ms (client timeout 1627 ms). Trace:
W20260808 01:11:33.727674 1604 rpcz_store.cc:269] 0808 01:11:32.104222 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.829123 (+724901us) service_pool.cc:227] Handling call
0808 01:11:33.592968 (+763845us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b5400304b54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:33.593152 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1243377920}
I20260808 01:11:33.729161 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.730311 1604 raft_consensus.cc:2379] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 7: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:33.730785 1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2437) took 1301 ms. Trace:
I20260808 01:11:33.731223 1604 rpcz_store.cc:276] 0808 01:11:32.428921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.728793 (+1299872us) service_pool.cc:227] Handling call
0808 01:11:33.730722 (+ 1929us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:33.443462 5168 raft_consensus.cc:504] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:33.620675 5312 consensus_meta.cc:287] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.570s user 0.013s sys 0.005s
I20260808 01:11:33.652006 1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2735) took 1143 ms. Trace:
W20260808 01:11:33.634042 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 95230us
W20260808 01:11:33.650115 5303 consensus_meta.cc:287] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.513s user 0.014s sys 0.003s
I20260808 01:11:33.631597 5122 raft_consensus.cc:3037] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:11:33.720228 4922 raft_consensus.cc:504] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
I20260808 01:11:33.745280 2209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49268 (request call id 1203) took 2480 ms. Trace:
I20260808 01:11:33.745517 1601 raft_consensus.cc:2454] T 82bb58b6f7c14c22814cfac6d2b2bf6d P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 8.
I20260808 01:11:33.745831 2209 rpcz_store.cc:276] 0808 01:11:31.264772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.206279 (+941507us) service_pool.cc:227] Handling call
0808 01:11:32.206586 (+ 307us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:32.610758 (+404172us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:33.744943 (+1134185us) spinlock_profiling.cc:243] Waited 361 ms on lock 0x7b54007f03d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:33.745201 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":829273472}
I20260808 01:11:33.747767 2209 raft_consensus.cc:1226] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:33.750449 1598 raft_consensus.cc:2379] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd in current term 8: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:33.686784 1519 raft_consensus.cc:2454] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 9.
I20260808 01:11:33.700702 1520 raft_consensus.cc:1226] T 6b1ad14dc361495f8e43a49a35526563 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 6. Current term is 7. Ops: []
I20260808 01:11:33.657778 5037 raft_consensus.cc:3037] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:11:33.712088 1594 rpcz_store.cc:276] 0808 01:11:32.504639 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.568827 (+1064188us) service_pool.cc:227] Handling call
0808 01:11:33.570352 (+ 1525us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:33.711494 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:33.699735 5319 raft_consensus.cc:3032] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:33.720747 1600 raft_consensus.cc:2354] T f6bb0551ce74437284392981d2921446 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7.
I20260808 01:11:33.732932 1604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12c84f631ee4ca7b405e28317db1b8a" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.756366 1517 raft_consensus.cc:2396] T 704a8e4165cc49079ada0c7993c21187 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260808 01:11:33.709487 1596 raft_consensus.cc:2454] T bc8c41cadc374229baf248366012f202 P a4862d0938c84fc4bd840e4c17acae78 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 8.
I20260808 01:11:33.677718 4887 leader_election.cc:288] T ca9890c6b79a40fcbbf70e44142a95d0 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:33.737982 1602 rpcz_store.cc:276] 0808 01:11:32.508533 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.508626 (+1000093us) service_pool.cc:227] Handling call
0808 01:11:33.651865 (+143239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:33.677335 2204 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 95b661b03f1d460b9657e61d4b0247cd [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 11.
I20260808 01:11:33.752902 1516 raft_consensus.cc:1261] T d80a0f28c7c94d45b7983fc0b1386caf P 6b09dbefd24b4d52b385c4dff5697dc5 [term 9 FOLLOWER]: Refusing update from remote peer a4862d0938c84fc4bd840e4c17acae78: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260808 01:11:33.756699 1604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2439) took 1327 ms. Trace:
I20260808 01:11:33.714345 1523 raft_consensus.cc:1226] T 212d8f7da4b84fae81802fe11cdd4b81 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 5. Current term is 6. Ops: []
W20260808 01:11:33.759284 1596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2435) took 1933 ms (client timeout 2330 ms). Trace:
W20260808 01:11:33.760013 1596 rpcz_store.cc:269] 0808 01:11:31.825935 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.542868 (+1716933us) service_pool.cc:227] Handling call
0808 01:11:33.759210 (+216342us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:33.721339 4953 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:33.740559 5227 raft_consensus.cc:686] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [term 8 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
I20260808 01:11:33.762130 1596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c8c7709e68463182371aa8224d2da2" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
I20260808 01:11:33.766599 1600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2732) took 1261 ms. Trace:
I20260808 01:11:32.602877 1568 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:45628 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2726)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2728)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2430)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2719)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2725)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 714)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 701)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2721)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 718)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 715)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2722)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2723)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2730)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2441)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2734)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2444)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2724)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2735)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 720)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2438)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2435)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2736)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2731)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2732)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2437)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2439)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2440)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 719)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2729)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2442)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2434)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2733)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2433)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2436)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2424)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2702)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2703)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2425)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 691)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2426)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2431)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 700)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 702)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 705)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 712)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 716)
I20260808 01:11:34.679615 5270 raft_consensus.cc:482] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:33.881979 1489 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 2.222s (SENT) [suppressed 40 similar messages]
W20260808 01:11:34.517068 1410 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.492s user 0.014s sys 0.015s
W20260808 01:11:34.681035 1410 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.656s user 0.017s sys 0.016s
W20260808 01:11:34.252655 1408 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.224s user 0.006s sys 0.013s
W20260808 01:11:34.126612 1525 consensus_meta.cc:287] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.513s user 0.012s sys 0.002s
W20260808 01:11:34.682154 1408 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.653s user 0.008s sys 0.020s
I20260808 01:11:34.135174 5062 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:33.996932 5101 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.201s user 0.022s sys 0.082s
I20260808 01:11:34.135399 5094 raft_consensus.cc:482] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:33.743112 5312 raft_consensus.cc:504] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } }
W20260808 01:11:34.561820 1518 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.547s user 0.004s sys 0.031s
I20260808 01:11:34.684677 1521 raft_consensus.cc:2354] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for earlier term 7. Current term is 8.
I20260808 01:11:33.968268 5185 raft_consensus.cc:482] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:34.685890 5099 raft_consensus.cc:482] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:34.692572 5185 raft_consensus.cc:504] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:34.693321 5099 raft_consensus.cc:504] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:33.957158 4887 raft_consensus.cc:482] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:33.968767 1448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60392 (request call id 922) took 1531 ms. Trace:
W20260808 01:11:33.997197 4871 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.943s user 0.027s sys 0.062s
W20260808 01:11:34.057724 2205 consensus_meta.cc:287] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 0.653s user 0.010s sys 0.002s
W20260808 01:11:33.809368 5088 consensus_meta.cc:287] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5: Time spent flushing consensus metadata: real 0.831s user 0.007s sys 0.010s
I20260808 01:11:33.863305 5019 leader_election.cc:288] T a3ce33a519294ae1bce66c8aca8cf601 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 election: Requested vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:33.847383 1517 raft_consensus.cc:1261] T bc54c3f2f5e04f63b88243c0d2d6babe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:11:34.048234 5133 raft_consensus.cc:482] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:34.229773 5260 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:33.796954 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba23837e3184c0d99703d8b5b0accbe" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:33.761452 5319 raft_consensus.cc:729] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3fb1a6380d27401a831cfe5f86a4cc9d, State: Running, Role: LEADER
I20260808 01:11:33.744700 5074 raft_consensus.cc:686] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [term 6 LEADER]: Becoming Leader. State: Replica: 95b661b03f1d460b9657e61d4b0247cd, State: Running, Role: LEADER
W20260808 01:11:33.956427 1522 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.926s user 0.034s sys 0.064s
W20260808 01:11:34.683661 5101 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.888s user 0.025s sys 0.091s
I20260808 01:11:33.759003 1604 rpcz_store.cc:276] 0808 01:11:32.429565 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.732603 (+1303038us) service_pool.cc:227] Handling call
0808 01:11:33.756622 (+ 24019us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:33.767444 1594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78" is_pre_election: true
W20260808 01:11:34.682989 1408 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.68:34571, user_credentials={real_user=slave}} blocked reactor thread for 1.66642e+06us
W20260808 01:11:34.078292 2202 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.981s user 0.012s sys 0.028s
W20260808 01:11:34.690743 2207 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1207) took 3118 ms (client timeout 1852 ms). Trace:
I20260808 01:11:34.690667 5208 raft_consensus.cc:482] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:34.682323 1525 raft_consensus.cc:2454] T ea62d4057ca44da4b68e6710052dfbe5 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:34.676555 1602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea0da159669492a90fc257d91f4ba5c" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
I20260808 01:11:34.678624 1523 raft_consensus.cc:1261] T 417bc7d1279d48e89c3599b26117aaf9 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:11:34.760308 1523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36560 (request call id 747) took 1010 ms. Trace:
I20260808 01:11:34.760859 1523 rpcz_store.cc:276] 0808 01:11:33.749402 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.760299 (+ 10897us) service_pool.cc:227] Handling call
0808 01:11:34.678066 (+917767us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:34.759916 (+ 81850us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:34.760231 (+ 315us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:34.345873 1597 consensus_meta.cc:287] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.713s user 0.006s sys 0.006s
I20260808 01:11:34.680577 5270 raft_consensus.cc:504] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:34.696200 4887 raft_consensus.cc:504] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:34.762358 1597 raft_consensus.cc:2454] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b661b03f1d460b9657e61d4b0247cd in term 7.
W20260808 01:11:34.763051 1597 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 720) took 2253 ms (client timeout 1593 ms). Trace:
I20260808 01:11:34.768872 1602 raft_consensus.cc:2379] T 9ea0da159669492a90fc257d91f4ba5c P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d in current term 6: Already voted for candidate a4862d0938c84fc4bd840e4c17acae78 in this term.
I20260808 01:11:34.773465 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:34.562029 5213 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.420s user 0.012s sys 0.015s
I20260808 01:11:34.695912 1600 rpcz_store.cc:276] 0808 01:11:32.505014 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.619161 (+1114147us) service_pool.cc:227] Handling call
0808 01:11:33.766514 (+147353us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:34.798549 1600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2729) took 2609 ms (client timeout 2291 ms). Trace:
I20260808 01:11:34.697451 2205 raft_consensus.cc:2454] T 36f2393bca80434dbdc089722b52edd1 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
W20260808 01:11:34.799703 2205 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49260 (request call id 849) took 1598 ms (client timeout 1757 ms). Trace:
I20260808 01:11:34.397871 5286 leader_election.cc:288] T b316a05b1f7448158ae76530ee0ee127 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:34.700132 5260 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:34.695807 5185 leader_election.cc:288] T 3f18b5317cac4512ad16a64664fab357 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:34.712903 5263 leader_election.cc:288] T 55fb799d94774eb18fed4dd8fdb5ac63 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 11 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:34.683005 5062 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: NON_VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: true } }
I20260808 01:11:34.701426 5074 consensus_queue.cc:237] T 212d8f7da4b84fae81802fe11cdd4b81 P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:34.701303 5319 consensus_queue.cc:260] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:34.705534 2202 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.608s user 0.020s sys 0.039s
W20260808 01:11:34.738327 5122 consensus_meta.cc:287] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78: Time spent flushing consensus metadata: real 0.998s user 0.000s sys 0.013s
I20260808 01:11:34.707417 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58547863eabc48eda1552ed4240f68d3" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "95b661b03f1d460b9657e61d4b0247cd"
I20260808 01:11:34.720546 5168 leader_election.cc:288] T 82bb58b6f7c14c22814cfac6d2b2bf6d P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:34.705006 1594 raft_consensus.cc:2454] T 45ac77bfbd0c4849bbda5cd2e754a927 P a4862d0938c84fc4bd840e4c17acae78 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in term 4.
I20260808 01:11:34.699645 5133 raft_consensus.cc:504] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:34.701509 1599 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2726) took 2804 ms (client timeout 1793 ms). Trace:
W20260808 01:11:34.696805 4871 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.645s user 0.033s sys 0.069s
I20260808 01:11:34.697489 5227 consensus_queue.cc:237] T 846d38a10c5846789f0b926b61905c0b P 95b661b03f1d460b9657e61d4b0247cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:34.678186 4922 leader_election.cc:288] T a539e94f6f744f20b65ea3fa4bbf8be4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:34.139288 5324 log.cc:924] Time spent T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d: Append to log took a long time: real 0.305s user 0.000s sys 0.002s
I20260808 01:11:34.696393 5099 leader_election.cc:288] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:34.684206 5094 raft_consensus.cc:504] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:34.397337 5189 raft_consensus.cc:482] T 846d38a10c5846789f0b926b61905c0b P 3fb1a6380d27401a831cfe5f86a4cc9d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:34.783396 4935 raft_consensus.cc:3032] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:11:34.769685 5208 raft_consensus.cc:504] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:34.786739 4887 leader_election.cc:288] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:34.788450 2200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 1098) took 3663 ms. Trace:
I20260808 01:11:34.789599 5270 leader_election.cc:288] T a668c2efd2dd4f5b946b70904c143c17 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:34.790446 2203 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 1115) took 2081 ms. Trace:
I20260808 01:11:34.796696 5312 leader_election.cc:288] T 58c8c7709e68463182371aa8224d2da2 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 8 election: Requested vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
W20260808 01:11:34.800168 2205 rpcz_store.cc:269] 0808 01:11:33.201749 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.403242 (+201493us) service_pool.cc:227] Handling call
0808 01:11:34.799640 (+1396398us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:34.252821 5174 raft_consensus.cc:482] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:34.399017 5241 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:34.300011 5147 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 election: Requested vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:34.335618 5089 leader_election.cc:288] T 43bc87d4fdb1436f875364cc78ec3fca P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:34.799338 5213 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.656s user 0.012s sys 0.017s
W20260808 01:11:34.706684 2207 rpcz_store.cc:269] 0808 01:11:31.571823 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.595147 (+ 23324us) service_pool.cc:227] Handling call
0808 01:11:34.690503 (+3095356us) spinlock_profiling.cc:243] Waited 413 ms on lock 0x7b5400482454. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:34.690654 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":950228352}
W20260808 01:11:34.813215 5258 consensus_meta.cc:287] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 1.649s user 0.016s sys 0.002s
W20260808 01:11:34.681417 1410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.212.66:37093, user_credentials={real_user=slave}} blocked reactor thread for 1.67291e+06us
W20260808 01:11:34.681170 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 799179us
I20260808 01:11:34.699610 1516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2268) took 1124 ms. Trace:
W20260808 01:11:34.697377 1518 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.683s user 0.005s sys 0.032s
I20260808 01:11:34.683003 1519 raft_consensus.cc:1226] T 921f3eb5150b4105a4b68355cca8cb7a P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Rejecting Update request from peer a4862d0938c84fc4bd840e4c17acae78 for earlier term 7. Current term is 8. Ops: []
W20260808 01:11:34.693338 1521 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58280 (request call id 2252) took 1286 ms (client timeout 1671 ms). Trace:
I20260808 01:11:34.698906 1596 raft_consensus.cc:2396] T 58c8c7709e68463182371aa8224d2da2 P a4862d0938c84fc4bd840e4c17acae78 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb1a6380d27401a831cfe5f86a4cc9d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260808 01:11:34.774446 1523 raft_consensus.cc:2379] T bc8c41cadc374229baf248366012f202 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 8: Already voted for candidate 6b09dbefd24b4d52b385c4dff5697dc5 in this term.
W20260808 01:11:34.701642 1522 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.672s user 0.043s sys 0.068s
W20260808 01:11:34.817695 1599 rpcz_store.cc:269] 0808 01:11:31.897477 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.826290 (+928813us) service_pool.cc:227] Handling call
0808 01:11:34.701348 (+1875058us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b5400209ed4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:34.701446 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":552947968}
W20260808 01:11:34.930946 2171 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.182s user 0.004s sys 0.029s
W20260808 01:11:34.931131 1599 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2442) took 2490 ms (client timeout 2070 ms). Trace:
W20260808 01:11:34.733757 1568 connection.cc:232] RPC call timeout handler was delayed by -2.0140049649926368s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1531s for a response
W20260808 01:11:34.763518 1597 rpcz_store.cc:269] 0808 01:11:32.509261 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.510347 (+1001086us) service_pool.cc:227] Handling call
0808 01:11:34.762946 (+1252599us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:34.992300 1597 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2448) took 1981 ms (client timeout 1525 ms). Trace:
W20260808 01:11:34.992796 1597 rpcz_store.cc:269] 0808 01:11:33.010604 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:34.991782 (+1981178us) service_pool.cc:214] Skipping call since client already timed out
0808 01:11:34.992036 (+ 254us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 01:11:34.994017 1597 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2434) took 3204 ms (client timeout 2762 ms). Trace:
W20260808 01:11:34.994431 1597 rpcz_store.cc:269] 0808 01:11:31.789905 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:34.993793 (+3203888us) service_pool.cc:214] Skipping call since client already timed out
0808 01:11:34.993954 (+ 161us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 01:11:34.995357 1597 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2432) took 3564 ms (client timeout 3177 ms). Trace:
W20260808 01:11:34.798974 1600 rpcz_store.cc:269] 0808 01:11:32.189071 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:34.798351 (+2609280us) service_pool.cc:214] Skipping call since client already timed out
0808 01:11:34.798472 (+ 121us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 01:11:34.791088 1602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2440) took 2361 ms (client timeout 2002 ms). Trace:
W20260808 01:11:34.996802 1600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2733) took 2488 ms (client timeout 2162 ms). Trace:
I20260808 01:11:34.996863 2202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49270 (request call id 1113) took 2492 ms. Trace:
W20260808 01:11:34.997303 1600 rpcz_store.cc:269] 0808 01:11:32.507994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:34.996466 (+2488472us) service_pool.cc:214] Skipping call since client already timed out
0808 01:11:34.996629 (+ 163us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 01:11:34.998592 1600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2433) took 3542 ms (client timeout 3251 ms). Trace:
W20260808 01:11:34.999238 1600 rpcz_store.cc:269] 0808 01:11:31.456357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:34.998368 (+3542011us) service_pool.cc:214] Skipping call since client already timed out
0808 01:11:34.998518 (+ 150us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260808 01:11:34.823694 2201 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49268 (request call id 1209) took 2988 ms (client timeout 2008 ms). Trace:
W20260808 01:11:35.000448 2201 rpcz_store.cc:269] 0808 01:11:31.835298 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:31.972256 (+136958us) service_pool.cc:227] Handling call
0808 01:11:34.823455 (+2851199us) spinlock_profiling.cc:243] Waited 446 ms on lock 0x7b540060d954. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:34.823625 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1026175360}
W20260808 01:11:34.703356 1604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 719) took 2194 ms (client timeout 1954 ms). Trace:
I20260808 01:11:35.002446 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e2d40d6b2943d883579bc4571ed4c1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:34.759377 1525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58280 (request call id 2265) took 1246 ms. Trace:
I20260808 01:11:34.698622 1524 raft_consensus.cc:2454] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in term 11.
W20260808 01:11:34.931715 2171 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.183s user 0.004s sys 0.029s
I20260808 01:11:35.032985 5237 raft_consensus.cc:482] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:35.033115 2171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.212.67:35675, user_credentials={real_user=slave}} blocked reactor thread for 1.28449e+06us
W20260808 01:11:34.933764 1599 rpcz_store.cc:269] 0808 01:11:32.440101 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:34.930539 (+2490438us) service_pool.cc:214] Skipping call since client already timed out
0808 01:11:34.930850 (+ 311us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 01:11:35.032636 1525 rpcz_store.cc:276] 0808 01:11:33.513616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.544077 (+ 30461us) service_pool.cc:227] Handling call
0808 01:11:34.759280 (+1215203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:35.037173 2205 raft_consensus.cc:1226] T eba23837e3184c0d99703d8b5b0accbe P 95b661b03f1d460b9657e61d4b0247cd [term 8 FOLLOWER]: Rejecting Update request from peer 3fb1a6380d27401a831cfe5f86a4cc9d for earlier term 7. Current term is 8. Ops: []
I20260808 01:11:35.037432 1599 raft_consensus.cc:1261] T bc9cccb55e8444778a9b2833e3d89a5a P a4862d0938c84fc4bd840e4c17acae78 [term 10 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260808 01:11:35.038096 1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2424) took 4667 ms. Trace:
I20260808 01:11:35.037778 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:34.927623 1523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58280 (request call id 2272) took 1166 ms. Trace:
W20260808 01:11:34.927606 1596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45628 (request call id 2446) took 2069 ms (client timeout 1587 ms). Trace:
I20260808 01:11:34.712401 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27b15da83cb42d49d1b0a703b5955ea" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:34.850095 5133 leader_election.cc:288] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3fb1a6380d27401a831cfe5f86a4cc9d (127.31.212.65:38291), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:34.923126 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:34.855343 5208 leader_election.cc:288] T 002e3bea322d445e8f5f8f09e2442db4 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:34.852927 5260 leader_election.cc:288] T 4e0a7f6aa072473098cb2c51da278a83 P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:34.850226 5220 raft_consensus.cc:482] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:34.861196 5094 leader_election.cc:288] T efd5468b2c6e4509ae17a98f9509ff03 P 95b661b03f1d460b9657e61d4b0247cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:34.834811 5319 raft_consensus.cc:3037] T bc54c3f2f5e04f63b88243c0d2d6babe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:34.819664 5103 raft_consensus.cc:482] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:34.987681 4887 raft_consensus.cc:482] T 5353a0d85aff48078011ca6efaf9ddfa P 3fb1a6380d27401a831cfe5f86a4cc9d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 95b661b03f1d460b9657e61d4b0247cd)
I20260808 01:11:34.717460 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdecb8f50de443095f848f2500fd0a8" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:34.922338 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5353a0d85aff48078011ca6efaf9ddfa" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:34.988970 5213 raft_consensus.cc:482] T bcdecb8f50de443095f848f2500fd0a8 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:34.920671 1439 raft_consensus.cc:1226] T eba23837e3184c0d99703d8b5b0accbe P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 LEADER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7. Ops: []
W20260808 01:11:34.926597 1521 rpcz_store.cc:269] 0808 01:11:33.413972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.414475 (+ 503us) service_pool.cc:227] Handling call
0808 01:11:34.693062 (+1278587us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54000c7e54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:34.693275 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2917897088}
I20260808 01:11:34.828512 2203 rpcz_store.cc:276] 0808 01:11:32.709257 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.509621 (+800364us) service_pool.cc:227] Handling call
0808 01:11:33.628459 (+118838us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:33.673120 (+ 44661us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:33.673183 (+ 63us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:33.673260 (+ 77us) raft_consensus.cc:1631] Marking committed up to 7
0808 01:11:33.673332 (+ 72us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:34.790168 (+1116836us) spinlock_profiling.cc:243] Waited 44.2 ms on lock 0x7b54004538d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:34.790225 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:34.790375 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":101695744}
W20260808 01:11:35.003129 1604 rpcz_store.cc:269] 0808 01:11:32.509013 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:34.703123 (+2194110us) service_pool.cc:214] Skipping call since client already timed out
0808 01:11:34.703246 (+ 123us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 01:11:34.829844 5174 raft_consensus.cc:504] T d80a0f28c7c94d45b7983fc0b1386caf P 95b661b03f1d460b9657e61d4b0247cd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
W20260808 01:11:34.815056 1594 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45638 (request call id 2717) took 3463 ms (client timeout 3026 ms). Trace:
I20260808 01:11:34.855129 5266 leader_election.cc:288] T 6b1ad14dc361495f8e43a49a35526563 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
W20260808 01:11:34.853751 5037 consensus_meta.cc:287] T a19c1759594d4722abadf576352f9a98 P 95b661b03f1d460b9657e61d4b0247cd: Time spent flushing consensus metadata: real 1.062s user 0.012s sys 0.002s
W20260808 01:11:35.005420 1601 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34182 (request call id 715) took 3175 ms (client timeout 2110 ms). Trace:
W20260808 01:11:34.925815 1489 leader_election.cc:334] T 73a11329d12c4ba4abd2b4e0e29816c7 P 6b09dbefd24b4d52b385c4dff5697dc5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675): Timed out: RequestConsensusVote RPC to 127.31.212.67:35675 timed out after 1.505s (SENT)
I20260808 01:11:34.848477 5062 leader_election.cc:288] T d80a0f28c7c94d45b7983fc0b1386caf P 3fb1a6380d27401a831cfe5f86a4cc9d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), a4862d0938c84fc4bd840e4c17acae78 (127.31.212.67:35675)
I20260808 01:11:34.830322 5241 raft_consensus.cc:504] T 58547863eabc48eda1552ed4240f68d3 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:35.008709 1600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87907e719e4c46c4a19015f3f3bba308" candidate_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a4862d0938c84fc4bd840e4c17acae78"
W20260808 01:11:34.995710 1597 rpcz_store.cc:269] 0808 01:11:31.431439 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:34.995187 (+3563748us) service_pool.cc:214] Skipping call since client already timed out
0808 01:11:34.995296 (+ 109us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260808 01:11:34.713500 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f1af2e7c504c919e34f12663452f2a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:34.715369 1440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8c41cadc374229baf248366012f202" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:34.717015 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:34.923959 1516 rpcz_store.cc:276] 0808 01:11:33.575480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.726847 (+151367us) service_pool.cc:227] Handling call
0808 01:11:33.752605 (+ 25758us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:33.758749 (+ 6144us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:34.699383 (+940634us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b540023c8d0. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ea09 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:34.699537 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2978688}
W20260808 01:11:34.996706 1602 rpcz_store.cc:269] 0808 01:11:32.429890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.758398 (+1328508us) service_pool.cc:227] Handling call
0808 01:11:34.791005 (+1032607us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:35.010303 2207 raft_consensus.cc:1261] T 1e59ebd6ca6845789ae50d410b699ea7 P 95b661b03f1d460b9657e61d4b0247cd [term 12 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260808 01:11:35.030993 1520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2393bca80434dbdc089722b52edd1" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:34.926091 1519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58280 (request call id 2271) took 1228 ms. Trace:
I20260808 01:11:35.090745 1519 rpcz_store.cc:276] 0808 01:11:33.698990 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.754143 (+ 55153us) service_pool.cc:227] Handling call
0808 01:11:34.682817 (+928674us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:34.925436 (+242619us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:34.925859 (+ 423us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:35.092556 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498a0710a54e491dbd1a79ce64033f62" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5"
I20260808 01:11:35.034523 5237 raft_consensus.cc:504] T 1e59ebd6ca6845789ae50d410b699ea7 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:35.038492 1599 rpcz_store.cc:276] 0808 01:11:30.370638 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:35.036261 (+4665623us) service_pool.cc:227] Handling call
0808 01:11:35.036871 (+ 610us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:35.037819 (+ 948us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:35.037974 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:34.715030 1447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704a8e4165cc49079ada0c7993c21187" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
W20260808 01:11:35.083913 1601 rpcz_store.cc:269] 0808 01:11:31.829449 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.436827 (+1607378us) service_pool.cc:227] Handling call
0808 01:11:35.004955 (+1568128us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b540018ced4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:35.005223 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":707653376}
W20260808 01:11:35.086851 1594 rpcz_store.cc:269] 0808 01:11:31.351209 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.767119 (+2415910us) service_pool.cc:227] Handling call
0808 01:11:34.814938 (+1047819us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:35.044486 5283 raft_consensus.cc:482] T 1355c134ca5f45b381fb2b12b1865adb P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a4862d0938c84fc4bd840e4c17acae78)
I20260808 01:11:35.045954 5086 raft_consensus.cc:3032] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:11:35.046620 5265 raft_consensus.cc:482] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:11:35.055989 2171 connection.cc:232] RPC call timeout handler was delayed by -1.2843128009990323s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1765s for a response
W20260808 01:11:35.104681 2171 connection.cc:232] RPC call timeout handler was delayed by -1.1880352470034268s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1593s for a response
I20260808 01:11:35.058256 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d1d748c8f4d44942ad2f2e57e9162" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
I20260808 01:11:34.696465 1448 rpcz_store.cc:276] 0808 01:11:32.437803 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.535101 (+ 97298us) service_pool.cc:227] Handling call
0808 01:11:33.968555 (+1433454us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b540002dad4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:33.968703 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":410669440}
I20260808 01:11:35.039192 1523 rpcz_store.cc:276] 0808 01:11:33.761336 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:34.773177 (+1011841us) service_pool.cc:227] Handling call
0808 01:11:34.927318 (+154141us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260808 01:11:35.039552 1596 rpcz_store.cc:269] 0808 01:11:32.857546 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.761787 (+904241us) service_pool.cc:227] Handling call
0808 01:11:34.927083 (+1165296us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:34.828038 2200 rpcz_store.cc:276] 0808 01:11:31.125360 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:32.154206 (+1028846us) service_pool.cc:227] Handling call
0808 01:11:32.171176 (+ 16970us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:32.296196 (+125020us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:32.296245 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0808 01:11:32.296299 (+ 54us) raft_consensus.cc:1631] Marking committed up to 6
0808 01:11:32.296344 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:34.788143 (+2491799us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b540073df54. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca3ff51 00007fe34ca3e99e 00007fe3519802e9 00007fe34b066684 00007fe34b066612 00007fe34b066541 00007fe34b0664bc 00007fe34b065772 00007fe34ae6994c 00007fe34ae68d86
0808 01:11:34.788206 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:34.788372 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":286911232}
I20260808 01:11:35.020347 5122 raft_consensus.cc:482] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:35.020903 5270 raft_consensus.cc:482] T 58547863eabc48eda1552ed4240f68d3 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:11:34.826798 4935 raft_consensus.cc:729] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6b09dbefd24b4d52b385c4dff5697dc5, State: Running, Role: LEADER
I20260808 01:11:35.086923 1442 raft_consensus.cc:3037] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:11:35.092793 1602 raft_consensus.cc:1226] T ec7eb407c2934b7f9db6fa7a3ea0fae5 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Rejecting Update request from peer 6b09dbefd24b4d52b385c4dff5697dc5 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260808 01:11:34.997507 2202 rpcz_store.cc:276] 0808 01:11:32.504169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:33.062434 (+558265us) service_pool.cc:227] Handling call
0808 01:11:33.062681 (+ 247us) raft_consensus.cc:1457] Updating replica for 1 ops
0808 01:11:33.062978 (+ 297us) raft_consensus.cc:1520] Early marking committed up to index 0
0808 01:11:33.062995 (+ 17us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0808 01:11:33.063311 (+ 316us) log.cc:842] Serialized 26 byte log entry
0808 01:11:34.812651 (+1749340us) raft_consensus.cc:1631] Marking committed up to 8
0808 01:11:34.812690 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:34.812782 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0808 01:11:34.996154 (+183372us) raft_consensus.cc:1671] finished
0808 01:11:34.996227 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0808 01:11:34.996636 (+ 409us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15390,"raft.queue_time_us":3540,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":4398,"thread_start_us":1715011,"threads_started":1,"wal-append.queue_time_us":1746434}
I20260808 01:11:35.099673 1600 raft_consensus.cc:3037] T 87907e719e4c46c4a19015f3f3bba308 P a4862d0938c84fc4bd840e4c17acae78 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:11:35.100447 1599 raft_consensus.cc:1261] T 4f0008d696f943a3aaa520463af63298 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Refusing update from remote peer 95b661b03f1d460b9657e61d4b0247cd: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:11:34.869051 5223 leader_election.cc:288] T 73a11329d12c4ba4abd2b4e0e29816c7 P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:35.100462 1520 raft_consensus.cc:3037] T 36f2393bca80434dbdc089722b52edd1 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:11:35.103783 1445 raft_consensus.cc:3037] T 498a0710a54e491dbd1a79ce64033f62 P 3fb1a6380d27401a831cfe5f86a4cc9d [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:11:35.110396 1517 raft_consensus.cc:2379] T 5a8d1d748c8f4d44942ad2f2e57e9162 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 in current term 8: Already voted for candidate 3fb1a6380d27401a831cfe5f86a4cc9d in this term.
I20260808 01:11:35.111548 1523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ac77bfbd0c4849bbda5cd2e754a927" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" is_pre_election: true
W20260808 01:11:35.128682 1524 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58266 (request call id 2308) took 5123 ms (client timeout 2988 ms). Trace:
W20260808 01:11:35.129168 1524 rpcz_store.cc:269] 0808 01:11:30.004702 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:30.004869 (+ 167us) service_pool.cc:227] Handling call
0808 01:11:35.128227 (+5123358us) spinlock_profiling.cc:243] Waited 4.59 s on lock 0x7b540035c0d4. stack: 00007fe349ae5033 00007fe349ae4f30 00007fe349443534 00007fe35314f8b4 00007fe35314f849 00007fe34c9c4471 00007fe34ca432ae 00007fe351980c27 00007fe34b068c04 00007fe34b068b92 00007fe34b068ac1 00007fe34b068a3c 00007fe34b067cf2 00007fe34ae6994c 00007fe34ae68d86 00007fe34ae6872f
0808 01:11:35.128453 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10550257024}
I20260808 01:11:35.133268 1523 raft_consensus.cc:2396] T 45ac77bfbd0c4849bbda5cd2e754a927 P 6b09dbefd24b4d52b385c4dff5697dc5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4862d0938c84fc4bd840e4c17acae78 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:11:35.133903 5047 consensus_queue.cc:237] T 704a8e4165cc49079ada0c7993c21187 P 3fb1a6380d27401a831cfe5f86a4cc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 5, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } attrs { replace: true } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:35.141065 1442 raft_consensus.cc:2454] T a27b15da83cb42d49d1b0a703b5955ea P 3fb1a6380d27401a831cfe5f86a4cc9d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 11.
I20260808 01:11:35.140697 5122 raft_consensus.cc:504] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:35.104343 5086 raft_consensus.cc:729] T 212d8f7da4b84fae81802fe11cdd4b81 P a4862d0938c84fc4bd840e4c17acae78 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a4862d0938c84fc4bd840e4c17acae78, State: Running, Role: LEADER
I20260808 01:11:35.142486 1602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45638 (request call id 2703) took 4764 ms. Trace:
I20260808 01:11:35.139302 1599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34182 (request call id 691) took 4744 ms. Trace:
I20260808 01:11:35.147715 1599 rpcz_store.cc:276] 0808 01:11:30.380042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:35.099371 (+4719329us) service_pool.cc:227] Handling call
0808 01:11:35.099942 (+ 571us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:35.122009 (+ 22067us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:35.123753 (+ 1744us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:35.150590 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "95b661b03f1d460b9657e61d4b0247cd" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:35.150384 4935 consensus_queue.cc:260] T eba23837e3184c0d99703d8b5b0accbe P 6b09dbefd24b4d52b385c4dff5697dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.9, Last appended by leader: 9, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:35.152289 5103 raft_consensus.cc:504] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } attrs { replace: true } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:35.152658 5220 raft_consensus.cc:504] T 4e0a7f6aa072473098cb2c51da278a83 P 95b661b03f1d460b9657e61d4b0247cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "a4862d0938c84fc4bd840e4c17acae78" member_type: VOTER last_known_addr { host: "127.31.212.67" port: 35675 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68" port: 34571 } attrs { promote: false } }
I20260808 01:11:35.153745 1521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36560 (request call id 749) took 1403 ms. Trace:
I20260808 01:11:35.154398 1603 raft_consensus.cc:2396] T a83a963773094d6db6495b7009fd8690 P a4862d0938c84fc4bd840e4c17acae78 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 95b661b03f1d460b9657e61d4b0247cd for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260808 01:11:35.154754 1438 raft_consensus.cc:2454] T 31f1af2e7c504c919e34f12663452f2a P 3fb1a6380d27401a831cfe5f86a4cc9d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4862d0938c84fc4bd840e4c17acae78 in term 7.
I20260808 01:11:35.155893 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b316a05b1f7448158ae76530ee0ee127" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d"
I20260808 01:11:35.156343 1594 raft_consensus.cc:1261] T 1ccc57605ae24bf4bacee4e4741bbe95 P a4862d0938c84fc4bd840e4c17acae78 [term 7 FOLLOWER]: Refusing update from remote peer 3fb1a6380d27401a831cfe5f86a4cc9d: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260808 01:11:35.156912 1594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 2431) took 3967 ms. Trace:
I20260808 01:11:35.157016 1438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cccb55e8444778a9b2833e3d89a5a" candidate_uuid: "a4862d0938c84fc4bd840e4c17acae78" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" is_pre_election: true
I20260808 01:11:35.157250 1594 rpcz_store.cc:276] 0808 01:11:31.189898 (+ 0us) service_pool.cc:170] Inserting onto call queue
0808 01:11:35.155402 (+3965504us) service_pool.cc:227] Handling call
0808 01:11:35.155923 (+ 521us) raft_consensus.cc:1457] Updating replica for 0 ops
0808 01:11:35.156695 (+ 772us) raft_consensus.cc:1683] Filling consensus response to leader.
0808 01:11:35.156852 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260808 01:11:35.160630 5122 leader_election.cc:288] T b3583c46ad2f4aa8875c06ef46cde18a P a4862d0938c84fc4bd840e4c17acae78 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6b09dbefd24b4d52b385c4dff5697dc5 (127.31.212.66:37093), 95b661b03f1d460b9657e61d4b0247cd (127.31.212.68:34571)
I20260808 01:11:35.160018 5265 raft_consensus.cc:504] T e354fdfbf2264912b39cd20be2cb19f8 P 95b661b03f1d460b9657e61d4b0247cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fb1a6380d27401a831cfe5f86a4cc9d" member_type: VOTER last_known_addr { host: "127.31.212.65" port: 38291 } } peers { permanent_uuid: "6b09dbefd24b4d52b385c4dff5697dc5" member_type: VOTER last_known_addr { host: "127.31.212.66" port: 37093 } } peers { permanent_uuid: "95b661b03f1d460b9657e61d4b0247cd" member_type: VOTER last_known_addr { host: "127.31.212.68"